SAM-Veteran: An MLLM-Based Human-like SAM Agent for Reasoning Segmentation
Authors: Tianyuan Du
Haopeng Li
Zhen Fan
Jiarui Zhang
Panwang Pan
Yang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: reasoning segmentation
multi-modal large language model
reinforcement learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Significant progress has been made in reasoning segmentation by combining multi-modal large language models (MLLMs) with the Segment Anything Model (SAM): the former excel in reasoning and vision–language alignment, while the latter offers powerful pixel-level understanding. However, current paradigms fall short in exploiting SAM’s strengths, especially the ability to support iterative mask refinement by interactive segmentation, a process that human users can naturally perform. To bridge this gap, we introduce  **SAM-Veteran**, an experienced mask-aware SAM agent capable of emulating human interaction with SAM via a reasoning-driven segmentation workflow that integrates (i) generating bounding boxes given image–query pairs for SAM input, (ii) proposing refinement points based on SAM-generated masks, and (iii) adaptively terminating the process. Aiming for this goal, we propose a multi-task reinforcement learning framework based on Group Relative Policy Optimization (GRPO), which enhances the MLLM’s abilities in textual grounding and mask comprehension. Furthermore, we introduce a dynamic sampling strategy tailored for generating both boxes and points to stabilize training. Extensive experiments across diverse datasets show that SAM-Veteran achieves human-like interaction with SAM and establishes new state-of-the-art performance on both in-domain and out-of-domain benchmarks.

Published as a conference paper at ICLR 2026
SAM- VETERN: AN MLLM-BASED HUMAN-LIKE SAM AGENT FOR REASONING SEGMENTATION
Tianyuan Du', Haopeng Li", Zhen Fan, Jiarui Zhang, Panwang Pan, Yang Zhang'
PICO
(dut:aanyuan, lihaopeng. hop, fanzhen. 0315, zhangjlarui.zjjp zjjr13,, panpanwang, zhangyang.. 0621 Jebyyedanceee.ooommmm com
ABSTRACT
Significant progress has been made in reasoning segmentation by combining multi-modal large language models (MILMs) with the Segment Anything Model (SAM): the former excel in reasoning and vision-language alignment, while the latter offers powerful pixel-level understanding. However, current paradigms fall short in exploiting SAM's strengths, especially the ability to support iter- ative mask refinement by interactive segmentation, a process that human users can naturally perform. To bridge this gap. introduce SAM- perienced aask-aware SAM agent capable of emulating human interaction with SAM via reasoning-driven segmentation workflow that integrates (i) generating bounding boxes given image-query pairs for SAM input, (ii) proposing refinement points based on --ne-aenertted masks, and (ii) adaptively terminating the pro- cess. Aiming for this goal, we propose a multi-task reinforcement learning frame- work based on Group Relative Policy Optimization (GRPO), which enhances the MLLM's abilities in textual grounding and mask comprehension. Furthermore, we introduce dynamic sampling strategy tailored for generating both boxes and points to stabilize training. Extensive experiments across diverse datasets show that AAM-Veteran achieves human-like interaction with SAM and establishes new state-of- kfate-of-the-ade the--ee-ttttt performance on both in-domain and out-of-domain benchmarks.
INTRODUCTION
Reasoning segmentation generates pixel-level binary masks by interpreting textual queries through logical reasoning (Yu al., 2018; Yang al., 2022; Wang al., 2022; Liu al., 2023b; Zou et al., 2023a;b; Yang et al., 2023; Lai et al., 2024; Ren et al., 2024a;b; Rasheed et al., 2024; Liu al., 2025a). Unlike traditional semantic instance segmentation, which depends on predefined categorical labels (e.g.. dog or baby), reasoning segmentation is designed to handle more com- plex, context-dependent queries such the object held by the person in red. This setting better reflects real-world needs for intelligent assistants and robots, while also imposing higher demands on the model. It requires advanced capabilities such as nuanced textual understanding and logical reasoning, fine-grained visual perception and image-texx alignment, and broad domain knowledge combined with common sense. Multi-modal large language models (MILMss) (Li et al., 2023; Liu et al., 2023a; Wang et al., 2024; Chen et al., 2024b; Liu et al., 2024; Yang et al., 2025; Bai et al., 2025), owing to their integrated reasoning and multi-modal perception abilities, have thus become a central component in the latest paradigms for this task.
Recent studies have investigated two primary MLLM-based paradigms for this task: (1) Supervised Fine-Tuning (SFT), where MLLMs generate special tokens that control learnable segmentation head or decoder, thereby enabling end-to-end training as a unified model (Yan et al., 2024; Lai et al., 2024; Yan et al., 2025); and (2) Reinforcement Learning (RL), where MILMSs are optimized with reward signals for generating boxes and/or points that are then fed into Segment Anything Model (SAM) (Kirillov et al., 2023) produce the final segmentation (Liu al., 2025b; Huang et al., 2025). While SFT-based methods effectively incorporate the reasoning capability of MLLMs into
"Fqual contribution. 'Corresponding Author.

---

Published as a conference paper at ICLR 2026
Initial Box Generation
Iterative Refinement
Termination
Pronpt
Mask.
Pronpt
Mast
Pronpt
bounding mnver qustion Toown sipu g
nask the arowe
for
andind p onitep ofle.
and res pilipillienn topict
SAW-Veteron
EAM-Veteran
SAM-Veteran
chbir white piloe'a
siick pillo
aitoo The green aug manaater rusk in the image
entap coult, Siee 20% PUSTE.24
segroit cittiite
SAM
Termina
Figure I: Inference workflow of SAM-Veteran. Given image-question pair, SAM-Veteran first predicts bounding box for the target, which SAM converts into an initial mask. SAM- Veteran iteratively refines this mask by generating refinement points for SAM, until either both points null 1. (indicating satisfactory segmentation) or the maximum refinement step is reached.
segmentation pipelines, they suffer from two key limitations: (i) catastrophic forgetting of general reasoning abilities, and (ii) poor generalization out-of-domain data (Chu al., 2025). RL-based methods, on the other hand, either decouple MLLMs from SAM during training, leading to sub- optimal segmentation inputs (Liu al., 2025a), or fail to fully exploit SAM's capacity for iterative refinement and interactive segmentation as human users (Huang et al., 2025). SegAgent (Zhu et al., 2025b) the first, and so far the only, approach that emulates the behavior of human SAM users, performing SAM-friendly point generation and iterative mask refinement through RL-style opti- mization. However, it depends on bounding boxes generated by another MLLM as the initial input for SAM and lacks the ability to adaptively terminate the refinement process as humans do. More- over, the reliance on hand-crafted ground truth for point traces may constrain its exploration of more effective strategies for mask refinement.
To address these limitations, we propose SAMM-eteran, an experienced MLLM-based SAM agent designed for human-like SAM usage. As illustrated in Figure 1, SAM-Veteran follows complete reasoning-driven segmentation workflow that includes: (1) generating bounding boxes given im- age-query pairs for SAM input, (2) proposing refinement points based on SAM-generated masks, and (3) adaptively terminating the process. This design closely mirrors the intuitive way humans interact with SAM. To enable the MLLM to perform this workflow, we propose multi-task rein- forcement learning framework based on Group Relative Policy Optimization (GRPO) (Shao et al., 2024), designed to enhance its capabilities in textual grounding and mask comprehension. The framework consists of: (i) textual grounding task, where the model learns to generate bounding boxes that maximize both box IoU and mask loU of SA.--predicted masks; (ii) a mask compre- hension task, where the model judges whether the SAM-predicted mask is satisfactory and, if not, generates refinement points to effectively improve mask IoU; and (iii) an auxiliary task, which further strengthens mask comprehension by requiring the model to identify the centers of flaws in manually corrupted masks with random false positives and/or false negatives. Moreover, to stabi- lize GRPO-based reinforcement learning, we incorporate the concept of dynamic sampling from DAPO (Yu et al., 2025) into our framework. Extensive experiments demonstrate that SAM-Veteran effectively generates bounding boxes for initial SAM segmentation, produces points for iterative mask refinement, and can adaptively terminate the process. Evaluated on multiple reasoning -baed segmentation datasets, it achieves of-t-off-the-ar (SOTA) performance on both in-domain and out- of-domain benchmarks. The contributions of this work are summarized as follows:
We propose SAM- Veteran, an experienced SAM agent that emulates human usage behavior through a reasoning-drivn driven segmentation workflow encompassing box generation, iterative mask refinement, and adaptive termination. To the best of our knowledge, it is the first model to unify all of these behaviors within a single framework.
We present a multi-task RL framework consisting of three tasks: grounding task, which trains the model to generate bounding boxes well-suited for SAM segmentation; mask comprehen-
2

---

Published as a conference paper at ICLR 2026
sion task, which enables the model to evaluate mask quality and provide refinement points when necessary; and an auxiliary task, which further enhances the model's mask comprehension ability.
We conduct extensive experiments on both in-domain and out-of-domain datasets. The results demonstrate that SAM-Veteran performs human-like SAM usage and achieves state-of-the-ar performance across multiple benchmarks.
RELATED WORK
Supervised Fine-Tuning Approaches. SFT has been widely adopted to adapt MLLMs for reason-
ing segmentation (Wei et al., 2024; Zhang et al.. 2024a; Bai et al., 2024; Sun et al., 2024; Zhang et al., 2024b; Yuan et al., 2024; Yan et al., 2025). Early efforts, such as LISA (Lai et al., 2024) and its extensions GSVA (Xia et al., 2024), PixeII.M (Ren et al., 2024b), and PerceptionGPPT (Pi ct al., 2024), employ SFT on mixed datasets containing mask labels to finetune large language mod- els (LLMs) utilizing LoRA (Hu et al., 2022) and segmentation modules jointly. Diverging from this paradigm, LLM-Seg (Wang & Ke, 2024) and HReasonSeg (Lin et al., 2025) reformulate the segmentation task as mask selection problem without the need to train the segmentation modules. These methods rely on implicit semantic tokens or spec-spec decoders to generate pixel-level masks, achieving strong performance in-domain benchmarks. MMCPF (Tang et al., 2024) and GenSAM (Hu et al., 2024) focus on camouflaged object segmentation with SAM by optimizing visual prompts, but neither realizes iterative refinement with mask comprehension.
Reinforcement Learning Approaches. RL has emerged as a promising alternative, enabling mod- els learn generalizable segmentation policies through reward signals (Wang al., 2025c; Wu et al., 2025; Wang et al., 2025b). Seg-Zero (Liu et al., 2025a) initializes its reasoning model with pretrained MILM and employs RL activate reasoning chains without requiring SFT on reasoning data. VisionReasoner (Liu et al., 2025b) broadens this scope by addressing multiple visual per- ception tasks, including detection, segmentation, and counting, within a unified GRPO framework. More recently, SAM-R1 (Huang et al., 2025) advances this line of research by directly integrating SAM into the RL feedback loop, where segmentation masks serve as fine- fine-graing sree-graine reward signals. Beyond purely RL-based methods with frozen segmentation modules, recent work explores hy- brid strategies that combine SFT and RL. For instance, POPEN (Zhu et al., 2025a) incorporates preference- based optimization align MILMs with human preferences via RL. Besides, SegAgent first generates synthetic trajectories using a modified RL based on StaR (Zelikman et al., 2022), and then finetunes the MLLMs on these trajectories mimic human interactions with SAM.
3 SAM-VETERAN
3.1 TASK FORMULATION
In this work, we empower the MLLM to mimic human behavior when using SAM for reasoning segmentation. The optimized MILM is treated experienced SAM user, referred as SAM- Veteran, which can carry out textual grounding (generate bounding boxes), iterative mask refinement (output point coordinates), and adaptive termination. Specifically, given an image I and a question Q, the MLLM is prompted with Q to generate a bounding box E Rd for the target object. This image and bounding box are then input to SAM, producing an initial segmentation mask M. Next, the image and the resulting mask are fed back into the MLLM, which is now prompted with Q" to generate refinement points: a positive point p² E R recover the false negative region and/or negative point p E TR2 to suppress the false positive region. These points, along with the previous mask, are passed to SAM to generate an updated segmentation M'. This iterative process continues until either the MILMM determines that the mask is satisfactory, at which it outputs null 1 for both points, or the maximum refinement step is reached (forced termination). This process can be framed as Markov Decision Process (MDP) A,, T, R),, where:
State E S: Represents the current segmentation result. Since the quality of the mask depends on the image and the question, we define the state as a triplet, i.e., = (M, I)Q).
Action a E A: Refers to the answer from the MLLM for SAM input. Specifically, the initial action is the bounding box for question grounding, i.e., b, while the subsequent actions
3

---

Published as a conference paper at ICLR 2026
Textual Grounding
Mask Comprehension
Auxiliary
Toxtinput
bourding dephant"
green
arrer green
5A nother siephont"
mother alephans"
Mui Mi-Mdal LLM
poruslet- (1 Shickee
preen ahe
SAM
Computing
Reward
Reward
Rewar
Rowarl
Formatting
Reward
SAM Reward
|||
Reward
Dectiton Reward
Point Rewardd
Figure 2: Multi-task RL framework comprising Textual Grounding, Mask Comprehension, and Auxiliary Mask Comprehension. Two rollouts (with their rewards) are shown in different colors (blue and yellow). In the final reward, different bar textures represent different reward functions.
are the points for mask refinement, i.e., E ((+ ,p",,,",, (x+, (null,))) When mask refinement is complete, the action
(null,nall),
Transition Function T S x A + S: Defines how the current state transitions to the next state given a specific action, i.c., = [(s, a). In our approach, the transition function based on segmentation mask M' given the image I, current mask M, and
SAM, which generates current action a from the MI..M..
S x S state and the action, and providing the reward for the current transition, i.e., = R(s', 8, a)... The details of the reward function are introduced in the following section.
Reward Function
Comprehensively considering the quality of current/next
Our goal is to learn policy To(a[s) for the MILMM that enables human-like SAM-using for reason- ing segmentation, by maximizing the expected reward [ R(s', 8, a)]. By this means, the optimized SAM--eteran is capable of using SAM to carry out complete reasoning segmentation workflow covering box generation, iterative mask refinement, and adaptive termination, emulating the intuitive, human-like logic of the process.
3.2 MULTI-TASK REINFORCEMENT LEARNING FOR SAM-VETERAN
To accomplish this workflow, SAM-Veteran relies on two key abilities: (1) textual grounding. which generates bounding box around the target, and (2) mask comprehension, which evalu- ates whether the SAM segmentation is satisfactory and, if not, produces refinement points to guide SAM in improving the mask. To equip the MLLM with these capabilities, we propose a Multi-Task Reinforcement Learning Framework for Veteran,, based on GPRO. as illustrated in Figure 2. Specifically, we provide the MLLM with task-specific prompts (detailed in Appendix A.1), prompt- ing generate multiple responses (rollouts), each including both the reasoning process and the
4

---

Published as a conference paper at ICLR 2026
final answer for the corresponding task. For each rollout, compute task-specific reward, which then used for GPRO optimization. The details of each task are outlined in the following sections.
3.2.1 TASK 1: TEXTUAL GROUNDING
This task reinforces the MLLM's ability generate a bounding box for the target. Specifically, the MLLM is prompted to locate the object in an image based on a given question. Formally, given pair of an image and a corresponding question Q, the MLLM receives soounding-speciiin prompt QB, which guides it to generate the reasoning process and bounding box coordinates b. We define three types of rewards for this task:
Box Reward: We define our box reward as the combination of box IoU reward Rio and box L reward I2, proposed in Seg-Zero. Given the GT box (GT E R2,, each component is defined as,
Rio
1,
(bU(b, (CT) 0.5
RE
1, Σ, 1b - 10
0, otherwise,
0, otherwise.
SAM Reward RSAM, We input the image and the bounding box b into SAM, which produces the segmentation mask M = SA)(T, The SAM reward RSAM is defined as the loU between the predicted mask M and the ground truth mask MGT i.e., RSAM IoU MT). SAM reward evaluates how well the box supports SAM's segmentation.
Formatting Reward: This reward enforces compliance with the required output format. In par- ticular, the MILM must encapsulate its reasoning process and final answer within predefined tags (1 score) and provide the answer in a strictly parsable format (1 score). This reward applies uniformly across all tasks.
3.2.2 TASK 2: MASK COMPREHENSION
This task trains the MLLM to generate points that serve additional inputs for SAM re- fine the mask, or to terminate the process if the mask already satisfactory. Specifically, we provide the MILM with an image over- laid by a green transparent mask M (following SegAgent), predicted by SAM, together with the corresponding question Q. The MLLM is then prompted with a refinement-specifie prompt Q", from which it predicts a positive point p and/or a negative point p The im- age 7, the mask M, and the refinement points (pt,2") fed into SAM, which outputs improved mask M' = SAM(1, M, pt p=) Regarding rewards, in addition to the SAM reward RSAM IoU( M GT) and formatting reward, we also incorporate decision reward and an IoU improvement (IoU imp.) reward for this task:
Table 1: IoU Imp. Reward.
(-1,0) (0, 0.1] (0.1, 0.5)
(0.5, I]
R
2
3
Table 2: Reward combinations in all cases.
Action a
Good Enough
Need Refinement
(pt.p7) pT)
mull,nul1) Others
RDCS +3 RENC
RDCS R
Decision Reward RDCS, This reward guides the MILMM terminate the refinement process
the appropriate time. Given a state 1,,,Q) two cases are considered: (1) the current mask is unsatisfactory and requires further refinement (Need Refinement), and (2) the mask is sufficiently accurate and should be accepted (Good Enough). The MLLM receives a score of if it produces refinement points * (null, null)) when the mask in the Need Refinement case, or terminates the process (a = (null, null)) when the mask is in the Good Enough case. In all other situations, the reward is set
IoU Imp. Reward RA: This reward evaluates how much the IoU is improved by the current action the mask needs refinement. is defined based on the loU change A IoU(M',MMTT IOU(M, MTT shown in Table 1. Specifically, the MILM receives the highest score of if the mask shows a significant loU improvement (Á 0.5); the reward is set to if the refinement process degrades the mask (A < 0).
The overall reward of this task is determined by case-action pairs. Table summarizes the combi- nation of the two rewards for each pair. For consistency, the maximum reward is fixed at in both
5

---

Published as a conference paper at ICLR 2026
the Need Refinement and Good Enough cases. In addition, an encouraging score of RENC 2 is granted when the model generates only one type of point in the Good Enough case.
A key challenge in this task lies in explicitly obtaining masks under the two designated scenarios. To collect the data, we first extract ground-truth (GT) bounding boxes for the target objects in the training set and then provide each box-image pair to SAM for segmentation. The resulting masks are categorized as Need Refinement if their IoU is low, and as Good Enough if their IoU is high. To reduce the impact of imperfect edge annotations in the GT masks and to ensure that the MLLM focuses on substantive segmentation errors, we adopt a modified IoU calculation that excludes near- edge pixels' Under this scheme, masks with IoU are classified as Good Enough, whereas those with IoU < 0.9 are deemed Need Refinement. Furthermore, we balance the ratio between these two categories to avoid training bias.
3.2.3 TASK 3: AUXILIARY MASK COMPREIIENSION
When trained solely on the two tasks introduced above, we observe that the MLLM continues refin- ing the mask indefinitely, even when the mask is already perfect. We hypothesize that this behavior arises from the MLLM's limited ability to interpret masks within images, likely due to insufficient exposure to such data during pretraining. To enhance the model's 's mask comprehension, we intro- duce an auxiliary task, where ground-trute -ruth (GT) masks are intentionally corrupted with artificial im- perfections, including random polygonal inclusions (serve as false positives) and exclusions (serve as false negatives). Specifically, the MILM is provided with the image overlaid with the imperfect mask M and the corresponding question Q. The same sefineient-specffic question Q" is used to prompt the model, guiding it to predict a positive point pt and a negative point p corresponding to the centers of the false negative and false positive regions, respectively. To enforce this behavior, we define the following rewards:
Decision Reward R This reward encourages the MILM to output the appropriate type of point accordingly. Specifically, when a false positive region exists, the model receives score of 1 for outputting p- E IR2 Conversely, if no false positive region is present, it receives score of 1 for outputting null. 1. An analogous reward scheme is applied to the false negative region.
This reward encourages the MLLM to generate points at the correct locations. Specifically, the model receives a score of I if the output p falls within a false positive region, and an additional score of if its distance from the center of that region is less than Ta = 50 pixels. An analogous reward scheme is defined for p².
Point Reward:
3.3 DYNAMIC SAMPLING
Dynamic Sampling, proposed in DAPO, addresses the issue of gradient vanishing in GRPO. The original dynamic sampling strategy keeps sampling until the batch is fully filled with samples whose accuracy neither 0 nor 1. We adapt and extend this idea the three training tasks described above. For the grounding task, we over-sample candidate boxes and then apply Non-Maximum Suppression (NMS) to remove duplicates. This allows multiple objects or locations within the image to be sampled, thereby increasing the diversity of rewards across rollouts and enhancing the effectiveness of the advantage function. For the mask comprehension and auxiliary mask comprehension tasks, actions can be categorized into four cases: (null, 1, rull), (pt, null), (null,p ). and (pt.p) We over-sample actions until all four cases are represented. This practice encourages the MILM to recognize the differences among possible actions. By ensuring diverse set of actions with varying rewards, optimization becomes more stable and effective.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Implementation Details. We adopt Qwen2.5-VL--7B-mmee (Bai et al., 2025) as our base MILM and SAM2-Large (Ravi et al., 2025) as the segmentation module. For training, we configure the
We employ the modified ToU for all loU-hased mask rewards ( (RSAM and R-) during training, while the original loU is retained in evaluation for fair comparison. The detailed computation is in Appendix A.3.

---

Published as a conference paper at ICLR 2026
Table 3: We compare IoU (%) of different MLLM-based methods (7B version) across both in- domain and out-of-domain datasets.
Out-of-Domain
In-Domain
Method
RensonSeg val
ReasonSeg lest
RefCOCO testA
RefCOCO+ testA
RefCOCOp test
gloU
cloU
gloU
cloU
cloU
cloU
cloU
Qwen2.5-LL+SAM2
57.2
41.4
53.0
48.1
76.1
71.4
64.8
Supervised Finetuning
LISA (Lai et 2024)
53.6
52.3
48.7
48.8
76.5
67.4
68.5
VISA (Yan ct al., 2024)
52.7
57.8
75.7
64.8
66.4
PixelIM (Ren al., 2024b)
76.5
71.7
70.5
PerceptionGPT
al., 2024)
78.6
73,9
71.7
GSVA (Xia al., 2024)
50.5
56.4
78.9
69.6
73.3
Reinforcement Learning POPEN (Zhu el ul.,, 2025u)
60.2
64.5
79.9
74.4
74,6
SegAgeni (Zhu et al., 2025b)
33.0
25.4
33.5
31.3
80.3
75.5
74.6
Seg-Zero (Liu al., 2025a)
62.6
62.0
57.5
52.0
80.3
76.2
72.6
SAM-RI (Huang al., 2025)
64.0
55.8
60.2
54.3
79.2
74.7
73.1
SAM Veteran
68.2
67.3
62.6
56.1
80.8
76.6
73.4
batch size to 16 alongside rollout number of 8. AdamW (Loshchilov & Hutter, 2017) (7 is exploited as the optimizer with the learning rate 10 10-6, weight decay 0.01, and KL coefficient 0.005. To stabilize the training, we adopt global batch normalization from REINFORCE+ (Hu et al., 2025) instead of local standard deviation in GRPO. The model trained for one episode with the verl (Sheng et al., 2025) framework on eight 96GB GPUs, taking about 30 hours. To balance effectiveness and efficiency in evaluation, we limit the refinement process to a maximum of steps unless specified. Following Seg-Zero and SAM-R1, 1, we resize all images to 840 x 840 for both training and evaluation. In addition, during each refinement step, we supply SAM with the initial bounding box obtained from grounding. More implementation details provided in Appendix A.2.
Datasets and Evaluation Metrics. Regarding datasets, we use ReiCOCOg (Yu et al., 2016) to train SAM-Veteral (more details in Appendix A.4) and evaluate it on RefCOCO(+/g) (in-domain) and ReasonSeg (Lai et al., 2024) (out-of-domain). Following Seg-Zero, we use gloU and cloU as the evaluation metrics, where gloU is the average of per-image cerrecction-ove--nnion (IoUs), and the cloU the cumulative intersection over the cumulative union.
4.2 COMPARISONS
We compare SAM-Veteran Veteran with both SFT-based and RL-based models, with results summarized in Table For Qwen2.5-VI. segmentation masks are obtained by directly passing its predicted bounding boxes into SAM. As shown in the table, SAM-Veteran outperforms all SFT-based methods on both out-of-domain and in-domain benchmarks. Compared with Seg-Zero, it achieves stronger results across all benchmarks, primarily because our framework integrates SAM's segmentation out- puts into the RL reward and enables iterative mask refinement. Although SegAgent reports higher scores on the RefCOCOg test set, its reliance on SFT with point trajectories limits its generalization ability, leading to poor performance out- outtoof-done of-domain data; furthermore, it requires more refinement steps and lacks adaptive termination. POPEN also achieves the best performance on RefCOCOg test; however, it relies on well-trained PixelL as its initialization and adopts a relatively complex multi-stage training pipeline. The previous oft-of-domain SOTA. SAM-RI, attains strong perfor- mance by adopting GRPO with SAM reward guidance. However, unlike A-M--an it does not support iterative refinement, resulting in performance gap. Overall, on in-domain datasets, SAM-reteran Veteran achieves performance comparable while on out-of-domain datasets, it consistently surpasses all baselines by a clear margin, underscor- ing the generalization benefits of reinforcement learning framework.
exceeding existing methods in terms of IoU,
4.3 ABLATION STUDY
Iterative Mask Refinement. Figure: illustrates the trends IoU across refinement steps. As shown, Seternteran demonstrates obvious trend of improving the loU compared to the initial

---

Published as a conference paper at ICLR 2026
ReasonSeg (g
100
RensonSeg test (gloU)
100
refCOCO testA (cloU)
100
refCOCOg testA (cloU)
100
Termination Ratio (%6)
243
4.4
8:5
6,29
R25
Jou (%)
n.p
00-11
-2
-2
0
0 A ToU (Ours)
2
2
(Qwen)
Termination Ratio (Ouns)
Figure 3: Trends of IoU (A) and termination ratio over refinement iterations.
Table 4: Ablation study on three training tasks: Textual Grounding (TG), Mask Comprehension (MC), and Auxiliary (A). We report the IoU along with the termination behavior of the models.
TO
MC
ReasonSeg val
ReasonSeg test
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
Avg.
Termination
Owen+SAM2
57.2
$3.0
76.1
71.4
64.8
64.4
Arbitrary
62.4
62.1
79.3
75.3
72.2
70.3
Arbitrary
67.4
62.5
80.6
76.2
73.6
72.1
Never
68.2
62.6
80.8
76.6
73.4
72.2
Adaptive
mask generated from a bounding box (step 0), with particularly notable gains on out-of-doman datasets. In addition, report the ratio of terminated samples each step. The figure shows that this termination ratio increases with iterative refinement, indicating that SAM-Veteran progressively considers more masks satisfactory and is able to adaptively terminate the process when appropriate. For the original Qwen model, the loU declines throughout the process, indicating insufficient mask comprehension to support effective refinement.
Multi-Task Training. Table 4 presents the impact of the three training tasks. For Qwen and our model trained solely on the textual grounding task, we report results based only on the predicted bounding boxes. Our pure grounding model outperforms Qwen by large margin, but both exhibit arbitrary termination behavior. Incorporating mask comprehension substantially improves accuracy; however, it leads to indefinite refinement in the absence of an effective termination policy. Introduc- ing the auxiliary task further strengthens semantic understanding and enables adaptive termination. When all three tasks are combined, the model achieves the highest overall IoU while maintaining adaptive refinement termination strategy.
Reward Design. Table 5 shows the results of the ablation study on reward design. Specifically, We study the model trained without SAM reward RSAM decision reward RDCS (RC and RDCS ) or loU improvement reward RAA. Besides, we also replace RA with hard version R = 1a>0- As the results show, removing either component leads to consistent performance drops across datasets, highlighting their complementary roles in guiding effective refinement. Moreover, Rt yields weaker results than the original design, confirming that RA more effective.
Dynamic Sampling and Chain of Thought. The ablation results in Table 6 show that both dynamic sampling (DS) and chain of thought (CoT) contribute to BAM-Veteran performance. Removing DS leads to noticeable drops across most benchmarks, indicating its role in enhancing robustness. Meanwhile, removing CoT in all tasks also degrades performance, particularly on ReasonSeg, while the variant without CoT in mask-related tasks performs better but still lags behind the full model. These results confirm that DS and CoT are complementary and jointly important for effectiveness.
Table Ablation study on reward design, including removing SAM reward RSAM decision reward RDCS and IoU improvement reward R and replacing Ra with a hard version R
Model
ReasonSeg ReasonSeg test
RefCOCO testA
Re/COCO+ testA
RefCOCOp test
Avg.
SAM- Veteran
68.2
62.6
80.8
76.6
73.4
72.2
w/o RSAM
67.0
60.7
80.4
76.1
72.2
71.3
w/o RDOS
64.1
60.6
80.4
75.7
72.1
70.6
w/o R
64.1
60.2
79.4
75.9
72.3
70.4
R.A. > R
67.5
61.9
81.0
76.2
73.0
71.9
8

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on dynamic sampling (DS) and chain of thought (CoT).
Model
ReasonSeg val
ReasonSeg test
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
Avg.
SAM-Veteran
68.2
62.6
80.8
76.6
73.4
72.2
w/o DS
66.8
60.88
79.3
75.4
71.8
70.8
w/ CoT
64.4
60.1
80.0
75.5
73.0
70.6
w/ Mask CoT
66.1
61.5
80.3
76.8
73.3
71.6
The drea that dipleys the thre.
bad with piltows.
caliat bage deos madgpe for ael Hiah
crme
This
on the Aaple
oeneen Coe paint Bot eac "Hee poirrt out
Foremost giy sweafshert.
exiiid fnc. the rapo cnciiio the insored
de distrollee coei bur
poicies miliee
Women nearect camero
shw pric
coiting
Vieewar
with bcchole uniitey
1M0L 7H06
"reg, will
anl oppuar
Figure 4: Examples of the workflow with SAM-Veteran. Veteran. each example, the first column presents the visual input each step, the second column shows the text response, and the third column displays the SAM prediction given the boxes/points. Positive/Neeative points are shown as s/x.
Visualization. Figure 4 presents examples of the segmentation workflow performed by SAM- Veteran. In each case, the first column shows the visual input at each step, the second column displays the SAM-Veteran response, and the third column provides the SAM prediction given the boxes/points. We illustrate three representative scenarios, corresponding to 0, 1, and 2 refinement steps. These visualizations demonstrate that Aet--rteran capable of human-likk like usage of SAM, including providing the target box, identifying segmentation errors, and terminating once the mask satisfactory. Moreover, the process is reasoning- driven, as the model generates a plausible chain of thought to guide the final prediction. More visualizations are in Appendix B.6.
CONCLUSION
We propose Seteran, MLLM-based agent designed to perform human-like SAM usage for reasoning segmentation. Given an image and a text query. SAM-Veteran executes a complete work- flow: (1) generating bounding box input to SAM for initial mask prediction, (2) iteratively producing refinement points that serve as additional inputs for mask refinement, and (3) adaptively terminating the process once the mask is deemed satisfactory. To equip the MLLM with these capa- bilities for the workflow, we develop a multi-task reinforcement learning framework that explicitly rewards accurate bounding textual grounding and mask comprehension. Quantitative evaluations and qualitative visualizations demonstrate that SAM-Veteral achieves state-of-the tet-the-an art performance on both in-domain and out-of-domain benchmarks, while also exhibiting human-like behavior in SAM-based reasoning segmentation.

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint
www.:2308.19666 1(2)3, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou, One token to seg them all: Language instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems, 37:6833-6888 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056 27066, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma and Ranjay Krishna. Blink: Multimodal large language models can but not perceive. In European Conference on Computer Vision, pp. 148-166. Springer, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 122:3, 2022.
Jian Hu, Jiayi Lin, Shaogang Gong, and Weitong Cai. Relax image-specifee prompt requirement in sam: A single generic prompt for segmenting camouflaged objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12511-12511 2024.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03266 2025.
Jiaqi Huang, Zunnan Xu, Jun Zhou, Ting Liu, Yicheng Xiao, Mingwen Ou, Bowen Ji, Xiu Li, and Kehong Yuan. Sam-rl: Leveraging sam for reward feedback in multimodal segmentation via reinforcement learning. Advances in Neural Information Processing Systems, 2025.
Donggon Jang, Yucheol Cho, Suin Lee, Tachyeon Kim, and Daeshik Kim. Mmr: A large-scale benchmark dataset for multi-target and multi-granularity reasoning segmentation. In The Thir- teenth International Conference on Learning Representations, 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference computer vision, pp. 4015 2023.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 61 16266, 2023.
10

---

Published as a conference paper at ICLR 2026
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Bohao Li, Yuying Ge. Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-pue --acc----puus Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404. 16790, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image image pre-tran pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Weihuang Lin, Yiwei Ma, Xiaoshuai Sun, Shuting He, Jiayi Ji, Liujuan Cao, and Rongrong Ji. Hrseg: High-resolution visual perception and enhancement for reasoning segmentation. arXiv preprint arXiv:2507. 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 34916, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296 26306, 2024.
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18653-
18663, 2023b.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a.
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiaya Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint arXiv:2505. 12081, 2025b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01101, 2017.
Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into 1lm. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 27124-27133, 2024.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33009-13018, 2024.
Nikhila Ravi, Valentin Gabeur, Suan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In The Thirteenth International Conference on Learning Representations, 2025.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024a.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoming with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint 200v:2240..03300, 2024.
11

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang. Siqi Ma, Qifan Wang. Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408..66: 088662 2024.
Ly Tang, Teng-Tao Jiang. Zhi-Hao Shen, Hao Zhang, Wei Chen, and Bo Li. Chain of visual perception: Harnessing multimodal large language models for zero-shot camouflaged object de- tection. In Proceedings of the 32nd ACM international conference on multimedia, pp. 8805-8814, 2024.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi- image understanding. In The Thirleenth International Conference on Learning Representations, 2025a.
Junchi Wang and Lei Ke. Llm-seg: Bridging image segmentation and large language model reason- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1765-1774, 2024.
Lingfeng Wang, Hualing Lin, Senda Chen, Tao Wang, Changxu Cheng, Yangyang Zhong, Dong Zheng, and Wuyue Zhao. Alto: Adaptive-length tokenizer for autoregressive mask generation. arXiv preprint arXiv:2505. 16495, 2025b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model's 's perception of the world at any resolution. arXiv preprint 2rXiv:2409.121999 2024.
Song Wang, Gongfan Fang, Lingdong Kong, Xiangtai Li, Jianyun Xu, Sheng Yang, Qiang Li, Jianke Zhu, and Xinchao Wang. Pixelthink: Towards efficient chain-of-pixt reasoning. arXiv preprint arXiv:2505.23777. 2025c.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11686-12 11655, 2022.
Cong Wei, Haoxian Tan, Yujie Zhong, Yujuu Yang, and Lin Ma. Lasagna: Language-based segmen- tation assistant for complex queries. arXiv preprint arXiv:2404. 08506, 2024.
Yonghuang Wu, Wenwen Zeng, Xuan Xie, Chengqian Zhao, Guoqing Wu, and Jinhua Yu. Sampo: Visual preference optimization for intent-aware segmentation with vision foundation models. arXiv preprint arXiv:2508.02464 2025.
xAI. Gr5k-1.5 vision preview. https: /xx www.aaa///wweeeeeemmeeeeeeeeees http///ooo--o 5v, 2024.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Gen- cralized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF 2024.
Conference on Computer Vision and Pattern Recognition, pp.
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models, In European Conference on Computer Vision, pp. 98-115. Springer, 2024.
Ziang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu Zeng, Zilei Wang, Yali Wang, Yu Qiao, et al. Task preference optimization: Improving multimodal large language models with vision task alignment. In Proceedings of the Computer Vision and Pattenn Recognition Conference, pp. 29880-29892. 2025.
12

---

Published as a conference paper at ICLR 2026
Rui Yang. Lin Song, Yicheng Xiao, Runhui Huang, Yixiao Ge. Ying Shan, and Hengshuang Zhao. Haplovl: A single-transfong baseline for multi-modal understanding. arXiv preprint 1rXiv:2503.14694, 2025.
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240. 2023.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 18155-18165, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg. and Tamara L Berg. Modeling context in referring expressions. In European conference computer vision, pp. 69 85. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1307-1315, 2018.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 2025.
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu.
Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 28202-28211, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476- 2022.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-Ilava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in neural information processing systems, 37:71737-71767, 2024a.
Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pp. 74-91. Springer, 2024b.
Lanyun Zhu, Tianrun Chen, Qianxiong Xu, Xuanyi Liu, Deyi Ji, Haiyang Wu, De Wen Soh, and Jun Liu. Popen: Preference-based optimization and ensemble for lvim-based reasoning segmentation. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 30231-30240 2025a.
Muzhi Zhu, Yuzhuo Tian, Hao Chen, Chunluan Zhou, Qingpei Guo, Yang Liu, Ming Yang. and Chunhua Shen. Segagent: Exploring pixel understanding capabilities in milms by imitating hu- man annotator trajectories. In Proceedings of the Computer Vision and Pattenn Recognition Con- ference, pp. 3686-3696, 2025b.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Yuan, et Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15116 15127, 2023a.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jian- feng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36: 36:19979-11782 19782, 2023b.
13

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Reproducibility: More Implementation Details
A.I Prompts Used
A.2 More Configuration Details
A.3 IoU Excluding Near-Edge Pixels
A.4 Training Data
More Experiments and Analysis
B.1
More Reasoning Segmentation Benchmarks
B.2 General MLLM Benchmarks
B.3 Model Scalability
B.4 Hyperparameter Analysis
B.5 Inference Complexity
B.6 More Visualization
B.7 Failure Cases
Limitations
D Use of Large Language Models
14

---

Published as a conference paper at ICLR 2026
A REPRODUCIBILITY: MORE IMPLEMENTATION DETAILS
A.1 PROMPTS USED
We illustrate our prompts for textual grounding, mask comprehension, and auxiliary mask compre- hension in Figure 5.
Textusl Grounding (2*)
Prompt for Mask Comprehension Auxiliary (Q")
Please point out any errors the green segmentation mask acconding
Let's out "(Question""
step by step.
question "(Questiont"
Oulput Hiininng protess chiinko t/thini and finsl answeer in contwars
answer canswers </antwor> tags. Output the center point of the error area 35ON formst. with the key "pos_polnn"" for the faiee negetive and poin!" for the false pasitive, Set the point to null if there are no errors. Le., cthinko thinking process here c/thinks
Output the thinking process cthinko tf
c/orowers hags. Output the bbox the torget object in 35ON format.
Here on exomple:
ctrinto thinking process here c/thinks
canswars ('bbox_2d" [xt, yl, x2, y2] c/ //answers
conswers 1 poin!_ null, "ningp Duyl c/answers
Figure 5: Prompts used for textual grounding, mask comprehension, and the auxiliary task.
A.2 MORE CONFIGURATION DETAILS
In our dynamic sampling strategy for bounding boxes, we apply Non-Maximum Suppression (NMS) with IoU threshold of 0.8 eliminate duplicates. For refinement points, we generate samples in counts of (1, 1) corresponding to the cases (null, null). (p* null), (null,p") ), and (p+ .P ). respectively. The pseudocode for dynamic sampling of bounding boxes and refinement points is provided in Algorithm I and Algorithm 2. To accelerate training, dynamic sampling is disabled after 300 iterations. Following DAPO, we increase the upper clipping bound to 0.28 to encourage exploration. During evaluation, repetition penalty of 1.1 is applied to reduce redundant token generation. Full configurations can be found Figure 6.
Algorithm 1: Dynamic Sampling for Box
Algorithm 2: Dynamic Sampling for Point
Input:
Actor model M, image I, text prompt P. number of rollouts R, maximum attempts A, maximum error count E, IoU threshold Output: Valid rollouts R
Input: Actor model M, image text prompt P. number of rollouts R, maximum attempts
Output: Valid rollouts R
R < Ø;
R +- 0, B + Ø, ne 0;
ne 0;
for a - 1 to A - do
cases < {error: 2, terminate 1, positive :
C - M(T, P), B
2, negative 2, both 1);
Pp < [B) ridx < (-1)
for a x to A - 1 do
C (1, P):
foreach rollout with index j in C do
foreach rollout r E C do if | R:
if [ = R: return R;
correctFormatt)) and n. < E
R RU (r), ne +1;
return R;
continue:
C t getCase(r); if cases/c] > 0:
B' - B' (getBox(F)):
R +- RU [r];:: cases[c] cases[c] -1;
Fidx + rias (j};
foreach h: E NMS(BB, [) R] do
if :: > ndp and [8] R:
if R 1 | > 0: C + ([I,
B - BU ( k}]:
R - RU [k]]:
R < R UC|: (R - [R]]]:
R - R > 0: L C- M(I, P), R RUCE (R (R- (R-IR))):
return R:
return R;
15

---

Published as a conference paper at ICLR 2026
data:
prompt length: 1300
weaponee length:
task: 'Textual Grounding"
train films: fata/refootog nask
repeat:
task: "Mack Comprshension"
files: 6ata/ref2000g 600
repeat:
task: "Aaxy"
train files: cata/ref30C0g 9c 840 nask
repeat:
algorithm:
grpo
use
true
worker:
actor:
global batch size: 16 use_xl_loss:
_oe_: 5.0e-3
clip low: 0,2
clip_high: 0.28 optim:
optimizer: Acamt ir: 1.00-6 weight_decay: 2.00-2
rollout:
temperature: 1.0
n: 8
dynamic sample:
nax_try:
stop_iter:
tasks:
task: "Cexcual Orounding*
meta infos:
meta_infos:
iou_tbr:
task:
Conprehension"
meta_infos,
required cases: [1, 2, 2,
max_error_cnt:
task: "huxiliary"
zequired cases: [1, 2, moxoorrox_ent:
zeward:
tasks:
"Textual Groundinc"
reword ("formacting revard", reward", reword"]
task: Compcelansson!
seward_list: [*formatting zavard", "san reward", "daciaton zeward", "iou Inp. reward")
task: "Auxiliary"
zeward_listt LofonatLing revard", "decision rwward" "point swward")
trainer:
total_opisodes:
Figure 6: Full configurations for the training of
A.3 IOU EXCLUDING NEAR-EDGE PIXELS
As mentioned in SegAgent, the masks in refCOCO(+/g) (Yu et al., 2016) were annotated with poly- gons, resulting in imperfect alignment with boundaries. So we refine the IoU computation in calcu- lating rewards by excluding pixels near the object boundaries, the Python code shown in Figure In addition, we also visualize the near-edge pixels computed from our algorithm in Figure
A.4 TRAINING DATA
For the grounding task, we follow Seg-Zero and use the same 9,000 samples from the full training set RefCOCOg. For the mask comprehension task, we also use these 9,000 samples, obtaining SAM predictions from the ground-truth bounding boxes. We then compute the edge-pixel-ex
are treated as Good Enough, while around 700 samples with loU < 0.9 are categorized as Need Refinement. For the auxiliary mask comprehension task, we randomly augment the 9,000 ground ground-truth masks by adding polygon-region inclusion exclusion, each applied independently with probability 0.5. The examples of data in the mask-related tasks are shown in Figure 9. To balance the data across tasks, upsample the mask comprehension data by repeating it five times during training.
IoU between the SAM masks and ground-truth masks. 600 samples with loU
16

---

Published as a conference paper at ICLR 2026
import numpy as np import cv2
def compute iou (prreppeeaaaee mask: np adaaray gt_mask: np, ndarray,,
smnore_edger int = 20) -> float:
pred_mask pred_mask. astype (bool)
gt_mask gt_mask. (bool)
Create ignore mask if ignore_edge 0:
Use Canny to detect edges
ysst_p edges - cv2.Canny (gt_mask. astype (np uint8) + 255, 100, 200)
kernel ones (ignore_eedie edge, ignore _gnore_edge), np. ap.uint8) ignore_mast mask dilate (edges, kernel,
iterations-l) astype (bool)
else:
ignore_man
np. like (gt_ mask,,,,, dtype=bool)
Apply ignore mask
valid salid_pred np. logical_aae
"ignore_mask)
valid gt ap www.oogggaaaai and (gt mask, "ignore mask)
intersection np. logicclllaannaacccoaaaa (valid_pred, valid_gt) sum union
np. lo.ioiical_or (valid pred, valid gt) sum
if union 0:
return # avoid NaN, both masks are empty return intersection / union
Figure 7: Python code compute IoU excluding pixels near the object boundaries.
GT Mask
Near-Edge Pixels
GT Mask
Near-Edge Pixels
Visualization of GT mask (green) and the corresponding near-edge pixels (blue). We exclude the near-edge pixels in loU computation for training process only.
Figure 8:
B MORE EXPERIMENTS AND ANALYSIS
B.1 MORE REASONING SEGMENTATION BENCHMARKS
To further demonstrate the effectiveness of SAM-Veteran, evaluated it two more reason- ing segmentation benchmarks, i.e., MMR (Jang et al., 2025) and MUSE (Ren et al., 2024b). For MMR, we adopt the object-only version for fair comparison. Because query in MMR may re- fer to multiple objects, we merge the corresponding instance masks and treat the merged mask as the ground-truth segmentation. For MUSE (test set), original setting involves multi-target and multi-referring segmentation, which differs from the widely used benchmarks. To align with our evaluation protocols, we convert each sample into a classic triplet image, single-object mask, and mask caption. We then feed the image and mask caption into the model and compare the predicted mask with the single-object ground-truth mask. Table 7 presents the results of different models on the two benchmarks. Qwen+SAM2 provides the baseline performance delivers a substantial improvement over this baseline. contrast, SegAgent performs poorly on MUSE and nearly fails MMR, reflecting the limited generalization ability introduced by its SFT-
both datasets. Seg-Zero
17

---

Published as a conference paper at ICLR 2026
Mask
Avxillory Mask Comprehension
Refinment
Good
(wulL
(P"
Figure 9: Examples of visual inputs for the task of mask comprehension and the task of auxiliary mask comprehension.
Table 7: Results on MMR and MUSE.
MMR (Jang et al., 2025)
MUSE (Wang et al., 2025c)
Method
gloU
cloU
gloU
cloU
SAM2+SAM2
33.18
26.63
43.28
45.79
Seg-Zero (Liu et al., 2025a)
37.91
29.44
52.16
54.38
SegAgent (Zhu al., 2025b)
13.34
16.56
37.29
43.48
SAM- Veteran
40.38
30.74
53.63
57.42
based training. Our SAM-Veteran further surpasses Seg-Zero by a clear margin on both benchmarks, demonstrating the effectiveness of our RL-based multi-task training framework.
B.2 GENERAL MILM BENCHMARKS
To substantiate our claim that the proposed RL framework mitigates catastrophic forgetting gen- eral reasoning ability, evaluate dard general-purpose MLLM benchmarks (results shown in Table 8). As observed, the RL-based methods, Seg-Zero and our SAM-Veteres maintain performance comparable their respective base MLLM (Qwen2.5-VLL, demonstrating that RL training preserves general reasoning capabil- ity. In contrast, the SFT-based model SegAgent exhibits clear decline in performance vision- language benchmarks relative to its base model, Qwen-VL-Chan VL-Chat (Bai et al., 2023). indicating significant loss of generalization ability. These results confirm that SFT-based training prone to catastrophic forgetting, whereas RL-based methods effectively avoid this issue.
with several baseline models-on stan-
general
B.3 MODEL SCALABILITY
To evaluate the scalability of our method, we scale the MLLM from 7B to 32B and conduct exper- iments Qwen2.5-ve VLL-3BB using the same settings the 7B model. To further test scalability in the refinement step. we increase the maximum refinement steps from 3 to 5 for the 32B model. The results of the 32B variant across different datasets presented in Table 9. As observed, the 32B version of SAM- Veteran achieves further improvements on most in-domain and out-of-domain datasets, confirming the scalability of approach.
18

---

Published as a conference paper at ICLR 2026
Table 8: Results of different models on general MLLM benchmarks. our environment.
means re-evaluation by in
Dataset
Qwen2.5-VL
Qwen2.5-VL
Beg-Zero'
SAM Veteran
Qwen- VL-Chat"
SegAgent
OCR-Related Understanding BEED-Benchh-2Pluss (Li al., 2024)
70.4
69.6
69.5
44,6
9.7
TextVQAa (Singh al., 2019)
84.9
85.3
85.4
84.2
60.2
1.29
General Visual Question Answering
(Chen al., 2024a)
29.0
MMF.um (Fu al., 2025)
2347
2503
2286
2328
1834
753
MUIRBench (Wang et al., 2025a)
$9.6
$8.3
57.4
$9.2
27.9
12.23
BLINK (Fu al., 2024)
55.3
14.4
4.42
RealWorldQA (cAl, 2024)
68.5
67.8
68.9
66.0
45.8
1.57
Table 9: Results of SAM- of 7B and 32B.
ReasonSeg val
ReasonSeg test
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
Model
gloU
cloU
gloU
cloU
cloU
cloU
cloU
SAM- VeteVeet-rnnB
68.2
67.3
62.6
56.1
80.8
76.6
73.4
SAM Veteran-322
72.3
70.0
62.9
58.2
80.4
77.4
73.4
B.4 HYPERPARAMETER ANALYSIS
We conduct more analysis hyperparamters follows, with the results shown in Table 10.
Task 1 R We explore different designs R in Task 1. Specifically, we evaluate several hard-threshold settings 0.3, 0.7, and the baseline 0.5-as as well soft variant defined as RB [bU(b, BCT). The results indicate that the soft formulation performs worse than all sard-threshold versions, and among the hard thresholds, 0.5 yields the best performance.
Task Reward. First, replace R² in Task 2 with linear variant whose maximum reward is 3, increasing linearly from 0 with respect to the IoU change A, i.e., R = 3 ReLU(A). Second, we the encouraging score RFNC Both modifications lead to slight decrease in performance compared to the baseline.
Task Td. We experiment with different values of Ta in Task 3 3-10, 30, and the baseline 50. The results show that тα = 50 achieves the best overall performance.
Training Rollout. For the training hyperparameters, we experiment with different numbers of roll- outs in GRPO-4, 16, and the baseline 8. The results show that using 8 rollouts yields the best overall performance.
B.5 INFERENCE COMPLEXITY
We compare the inference- cettennce--ieeeeee efficiency different models using two metries: the average number of MLLM inference steps and the average time cost sample on the RefCOCO testA dataset. The results are shown in Table 11. For Qwen, we report the results of generating boxes for SAM. For
Table 10: Results of different hyperparameter configurations.
Parameter
Value
ReasonSeg val
ReasonSeg
RefCOCO testA
RefCOCO+ testA
RefOOCOg
Baseline
68.2
62.6
80.8
76.6
73.4
72.3
Task 1 R
Hard 0.3
67.1
61.2
80.6
77.3
72.5
71.7
nA ReLU(A)
68.1
61.5
80.2
77.0
73.2
72.0
DENC D
66.8
62.0
80.2
76.6
72.9
71.7
Task 3
80.3
30
68.3
61.8
79.9
76.4
72.8
71.8
Rollout
62.0
79.9
76.1
72.0
71.3
16
66.4
61.9
80.3
76.7
73.4
71.7
19

---

Published as a conference paper at ICLR 2026
Table 1I: Inference cost comparison of different MLLMs.
Method
Inference Backend
Average Step
Average Time (s)
Qwen+SAM2
Transformers Transformers
3.11
SegZero (Liu et al., 2025a)
1
3,43
SegAgent (Zhu et al., 2025b)
Transformers
8,95
Transformers
2.08
5.09
SAM-Veteran
vLLM
2.21
2.47
Seg-Zero, the MILM outputs both the bounding boxes and the points for SAM in single step, whereas SegAgent adopts a fixed number of 7 refiement iterations for mask prediction.
As shown in the results, Qwen and Seg-Zero finish the task in a single step (about 3s), but their segmentation performance is inferior that of multi-step refinement methods, evidenced in Ta- ble 1. SegAgent, on the other hand, requires a fixed 7-step MILM inference pipeline (about 9s), leading to low efficiency. Our SAM-Veteran achieves stronger performance with substantially fewer steps (< 2.5 steps on average, about 5s each sample). Our method is slower per sample per step than SegAgent because CoT introduces more response tokens, resulting in improved performance at the cost of extra time. Nevertheless, our method strikes a more favorable balance between seg- mentation quality and inference efficiency. Furthermore, Transformers backend with the vLLM (Kwon et al., 2023) backend, which significantly reduces the inference aime--pproximatell cutting the time consumption in half. This optimization is applied to all evaluations in our experiments.
improve efficiency replacing the
B.6 MORE VISUALIZATION
More visualization of SAM- Neteran is provided in Figure 10. For each example, we show the orig- inal image, the initial SAM output from a bounding box, and the masks refined iteratively with points. As the figure demonstrates, SAM- can accurately locate targets based on the posed question. When the initial segmentation is imperfect, it identifies the error regions, and with addi- tional points as guidance, SAM produces progressively improved masks. Remarkably, the model capture not only obvious omissions, such as boxes errors, like imperfect segmentation on a fork and the lawn mower.
shelf, but also subtle, hard-to-detect
As SegAgent also adopts multi-step mask-refinement framework for reasoning segmentation, compare its prediction workflow with that of our SAM-Veteran in Figure 11, using both in-domain and out-of-domain examples. As illustrated, SegAgent performs fixed seven refinement steps and sometimes generates ineffective or irrelevant points. In contrast, SAM-Veteran consistently generates accurate bounding boxes and refinement points, while also being able to dynamically determine when to terminate the procedure. These qualitative results further highlight the advantages of our method over SegAgent.
B.7 FAILURE CASES
We present several failure cases of Veteran in Figure 12, categorized into four representative types in different colors. The blue examples illustrate grounding deficiencies, where the model either localizes the wrong target fails to capture certain instances. The orange examples highlight confusion with green objects or regions the image, which resemble the green mask. The green examples demonstrate cases where the model generates redundant wrong points. Finally, the yellow example reveals flaw in the reasoning process, where the model incorrectly interprets the green mask part of human outfit.
C LIMITATIONS
While SAM-Veteran is capable of performing general human-like SAM usage, it does not yet sup- port fully free-form behavior. For example, cannot generate new bounding boxes after the initial step revoke previous actions when the mask quality degrades. Enabling such capabilities would
20

---

Published as a conference paper at ICLR 2026
require sophisticated action space and additional data, which we leave for future work. More- over, following SegAgent, we overlay the mask on the image as imput to the MLLM. This changes the object's color and may lead performance degradation color-sensitive queries. In future work, we plan to investigate presenting segmentation results to the MILMSs in a more effective way.
D USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, we employed Large Language Model (ChatGPT) solely writing assistant. Its
restricted to refining grammar, improving sentence structure, and enhancing the clarity and readability of the text. All methods, claims, experimental results, and conclusions were conceived and developed exclusively by the authors.
21

---

Published as a conference paper at ICLR 2026
cor wthh catr that doee Sariet CoN
The obwer zhat zan pectec me scel and wexenc x Juon geting stiret
Agentionnn wearhg plana dnng necine
Loplos right that whiti
SOMIC
SONIC
5OMNC
Meintarsyy grosned
The lod' prassfast on the rall.
reguter citting of the arma Vintr abWe perper onald ussT for thur
cof KM
Wimton on heoiing shane
Malk
conmen the poor wfhen porpureddd
te pictare the alrecty
Figure 10: Examples of the segmentation workflow performed by SAM-Veteran. Each example shows the original image, the initial SAM output from a bounding box, and the masks iteratively refined with points. Different number of refinement steps (0, 1, and 2) are in different colors.
22

---

Published as a conference paper at ICLR 2026
finun your busines buthroo wont store towel, you put
Laft
efter day's for
might
Comparison of the segmentation workflows of SAM- Veteran and SegAgent. The blue and orange flows correspond to SAM- Veteran and SegAgent, respectively. The two examples on the left out-of-domain cases, while the two the right in-domain. Negative points of SegAgent are shown as blue dots e.
Figure 11:
23

---

Published as a conference paper at ICLR 2026
boy cating picc with chooe.
pertee onyre bildig hufo fhr holty aneert
carouass (3bos 38 po 4lL aill-rrmmmn
dhsnpha sra kon wifh Cimeree buad oieere up wnet oolo grown
Jrom coee vngaemenn spuiee COnAAET SOw. "rag.oeett
Figure 12: Failure of SAM- Veteran. Different types of emrors highlighted in different
colors. The GT mask is shown in purple.
24


Reviews:
[{"review_id": "gaSNJaVHeS", "reviewer": "Reviewer_yXBo", "summary": "This paper introduces SAM-Veteran, a multi-modal large language model (MLLM)-based agent designed to emulate human-like interaction with the Segment Anything Model (SAM) for reasoning segmentation. The key contributions include:  \n\n- A **multi-task reinforcement learning (RL) framework** that trains the MLLM to generate bounding boxes, refine masks iteratively via points, and adaptively terminate the process.  \n- A **dynamic sampling strategy** to stabilize training by diversifying box/point generation.  \n- State-of-the-art performance on in-domain (RefCOCO) and out-of-domain (ReasonSeg) benchmarks, demonstrating improved generalization over existing supervised fine-tuning (SFT) and RL-based methods.", "questions": "**Q1**: How does SAM-Veteran compare to SegAgent in terms of refinement steps and termination accuracy? The authors should include a direct comparison to clarify their method’s advantages.  \n**Q2**: Could the inference time and training requirement be provided? This would potentially enhance the feasibility of real-world deployment.\n\n**Q3**: The paper states that SAM-Veteran avoids \"catastrophic forgetting\" seen in SFT methods. Is there empirical evidence (e.g., performance on general MLLM benchmarks) to support this claim?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The work creatively combines MLLMs with SAM’s interactive segmentation capabilities, addressing a gap in prior RL-based methods that neglect iterative refinement. The integration of three RL tasks (grounding, mask comprehension, and auxiliary) to mimic human-SAM interaction is novel.  \n**Quality**: The experiments are thorough, including ablation studies on reward design, multi-task training, and dynamic sampling. The comparison with SFT and RL baselines (e.g., Seg-Zero, SAM-R1) validates the framework’s effectiveness.  \n**Clarity**: The paper is well-structured, with clear task formulations (e.g., MDP definitions) and visualizations of the workflow. The appendices provide implementation details, prompts, and failure case analysis.  \n**Significance**: The work advances MLLM-based segmentation by enabling human-like SAM usage, which could inspire future research on interactive vision-language systems. The code and configuration details (Figure 6) enhance reproducibility.", "weaknesses": "**Limited comparison with SegAgent**: While SegAgent (Zhu et al., 2025b) is mentioned, the paper does not quantitatively compare SAM-Veteran against it, despite both addressing iterative refinement. This omission weakens the claim of novelty.  \n\n**Inference time:** The paper does not provide inference time comparisons with baseline methods, which is critical for evaluating practical deployment feasibility. Additionally, detailed training resource requirements (e.g., GPU hours, memory consumption) are not explicitly reported, limiting reproducibility assessments and computational cost analysis for researchers with constrained resources.\n\n**SAM dependency**: The framework heavily relies on SAM for mask generation and reward computation. The impact of SAM’s inherent limitations (e.g., failure modes on fine-grained objects) on SAM-Veteran’s performance is not discussed.  \n**User study absence**: While the paper emphasizes \"human-like\" behavior, no user study evaluates whether the termination policy or refinement steps align with human preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8xh8lvTNf", "reviewer": "Reviewer_GtqV", "summary": "This paper presents SAM-Veteran as an MLLM-based solution for reasoning segmentation, of which the key feature is to emulate the human SAM users to refine the mask iteratively. Instead of relying on supervised fine-tuning of the MLLM, SAM-Veteran is built on top of the current RL-based training paradigms. The paper proposes to divide the iterative reasoning segmentation task into three sub-tasks and design reward models to help train the MLLM on each of those sub-tasks. The proposed method shows strong results on several reasoning segmentation benchmarks compared to prior SFT and RL-based methods.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, well-organized, and easy to follow. In general, the proposed method shows noticeable improvement over the prior RL-based methods and SFT-based methods, and is closer to actual human interaction with the SAM-like segmentation models. The paper includes a thorough experiment for analyzing the contribution of each design choice, quantitative results, and qualitative visualizations to clearly show the strength and potential failing cases of the proposed method.", "weaknesses": "The motivation for having an auxiliary task is clear; however, the improvement is not significant, as shown in Table 4. Meanwhile, this means that there is room for improvement in task 2. A better reward design or pipeline for mask comprehension is needed. This weakness, as the paper points out, is linked to the limited ability of MLLM to understand images with masks, and adding a specific type of color mask on the visual input directly will inevitably compromise the valuable information contained in the target region.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQ2MVS9SgK", "reviewer": "Reviewer_yz1X", "summary": "This paper proposes SAM-Veteran, an MLLM-based agent that mimics human interaction with Segment Anything Model, enabling a complete “box generation to iterative refinement to adaptive termination” reasoning segmentation workflow. Built on a multi-task RL framework using Group Relative Policy Optimization, the method enhances textual grounding and mask comprehension, achieving new state-of-the-art performance and strong cross-domain generalisation.", "questions": "1. Training stability. Given the multi-task RL design, how stable is training in practice? Are there signs of task interference or convergence issues?\n\n2. Generalisation & scalability. How well does the method scale with more refinement steps or larger models, and can it generalise to broader segmentation scenarios beyond the tested datasets?\n\n3. Hyperparameters & ablation. Many hyperparameters are manually set, but their selection process and sensitivity are unclear. Could the authors provide ablation or sensitivity analyses to justify these choices?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive experiments. The authors conduct extensive evaluations across multiple in-domain and out-of-domain datasets, providing solid empirical evidence for their claims.\n\n2. Clear visualisations. The workflow and results are illustrated with intuitive and well-structured figures, making it easy to understand the model’s behaviour.\n\n3. Well-organised and easy to follow. The paper is logically structured, with each component of the method introduced and justified clearly.\n\n4. Strong motivation and validation. The motivation is well grounded, and the proposed design is effectively validated through both quantitative results and qualitative analysis.", "weaknesses": "1. Missing related works. Some relevant literature, such as recent works on task-generic promptable segmentation[1][2] and Grounding SAM[3], is not discussed or compared in the related work section. This weakens the contextual positioning of the contribution.\n\n2. Insufficient hyperparameter analysis. Many components involve manually set hyperparameters (e.g., the threshold of R_{iou}^B=\n0.4), but the paper does not explain how these values were chosen or their sensitivity.\n\n3. Potential training complexity and stability issues . The framework integrates multiple tasks simultaneously, which may lead to excessive complexity. The paper does not provide sufficient analysis on training stability or convergence behaviour.\n\n[1] Hu, Jian, et al. \"Relax image-specific prompt requirement in sam: A single generic prompt for segmenting camouflaged objects.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 11. 2024.\n\n[2] Tang, Lv, et al. \"Chain of visual perception: Harnessing multimodal large language models for zero-shot camouflaged object detection.\" Proceedings of the 32nd ACM international conference on multimedia. 2024.\n\n[3] Ren, Tianhe, et al. \"Grounded sam: Assembling open-world models for diverse visual tasks.\" arXiv preprint arXiv:2401.14159 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ctjzbIUU91", "reviewer": "Reviewer_fRUK", "summary": "This paper introduces SAM-Veteran, a reasoning-driven segmentation framework that enables human-like interaction with the Segment Anything Model (SAM). The method integrates a multi-modal large language model (MLLM) with SAM to perform iterative segmentation through three key steps: generating bounding boxes from image–query pairs, proposing refinement points based on SAM-generated masks, and adaptively terminating the process. To train the SAM-Veteran, diverse reward functions are utilized. Extensive experiments demonstrate that SAM-Veteran achieves state-of-the-art performance across both in-domain and out-of-domain datasets, effectively leveraging SAM’s interactive segmentation strengths.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents the first unified framework that integrates bounding box generation, iterative mask refinement, and adaptive termination into a single reasoning-driven segmentation process.\n- It introduces a well-designed multi-task reinforcement learning framework to effectively train the SAM-Veteran workflow, enhancing both textual grounding and mask comprehension.\n- The proposed method achieves state-of-the-art performance on both in-domain and out-of-domain datasets, demonstrating strong generalization and robustness.", "weaknesses": "- Recently, in the reasoning segmentation task, several datasets beyond ReasonSeg have been proposed. (MUSE [1] for multi-target cases and MMR [2] for part-level reasoning). It would be interesting to see whether SAM-Veteran demonstrates good generalization performance across these diverse reasoning segmentation scenarios.\n\n[1] Ren, Zhongwei, et al. \"Pixellm: Pixel reasoning with large multimodal model.\" CVPR  2024.\n\n[2] Jang, Donggon, et al. \"Mmr: A large-scale benchmark dataset for multi-target and multi-granularity reasoning segmentation.\" ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)