PathChat-SegR1: Reasoning Segmentation in Pathology via SO-GRPO
Authors: Zelin Liu
Dongdong Chen
Yusong Sun
Yuqi Hu
Huang Jie
Sicheng Dong
Xu Han
Hongmei Yi
Qiyuan Bao
Lichi Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Clinical Reasoning
Reinforcement Learning
Reasoning Segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Segmentation in pathology image requires handling out-of-domain tissue morphologies and new pathologies beyond training distributions, where traditional closed-set segmentation approaches fail to generalize. 
Reasoning segmentation enables zero-shot generalization via prompting with text queries. 
However, existing reasoning segmentation models face three barriers when applied to pathology: 
(1) the vision encoder lack pathology-specific knowledge and robustness to staining variations, 
(2) the large language model (LLM) backbone for reasoning fails to identify whether it has gathered sufficient semantic context to trigger the segmentation output,
and (3) no reasoning segmentation benchmarks and datasets exist for pathology analysis. 
Consequently, we introduce PathChat-SegR1, a reasoning segmentation model built upon pathology-specific vision encoders trained with a novel stain-invariant self-distillation for robust pathology image representations. 
Moreover, we propose Segmentation-Optimized GRPO (SO-GRPO), a reinforcement learning method specifically for reasoning segmentation that learns to determine optimal segmentation timing based on accumulated reasoning context.
Finally, we construct a pathology-specific reasoning segmentation benchmark of 118,667 triplets of pathology image, ground-truth mask, query, and reasoning chain including both public and private pathology images. 
Zero-shot evaluation on pathology images with out-of-domain morphologies/pathologies shows 61\% improvement over state-of-the-art segmentation models.

Published as a conference paper at ICLR 2026
PATHCHAT-SEGRI: REASONING SEGMENTATION IN PATHOLOGY VIA SO-GRPO
Zelin Liu' Dongdong Chen'r, Yusong Sun', Yuqi Hu', Huang Jie?. Sicheng Dong', Xu Han', Hongmei Yi², Qiyuan Bao Lichi Zhang Shanghai Jiao Tong University SShaggaii Ruijin Hospital
ABSTRACT
Segmentation in pathology image requires handling out-of-domain tissue mor- phologies and new pathologies beyond training distributions, where traditional closed-set segmentation approaches fail to generalize. Reasoning segmentation enables zero-shot generalization via prompting with queries. However, ex- isting reasoning segmentation models face three barriers when applied to pathol- ogy: (1) the vision encoder lack caahology-specinee knowledge and robustness to staining variations, (2) the large language model (LLM) backbone for reason- ing fails identify whether it has gathered sufficient semantic context to trigger the segmentation output, and (3) no reasoning segmentation benchmarks and datasets exist for pathology analysis. Consequently, we introduce PathChat-SegR1. reasoning segmentation model built upon pathology-spede vision encoders trained with a novel stain-invariant self-distillation for robust pathology image representations. Moreover, we propose Segmentation- -ppittiio---ttiiiieedeeee GRPO (SO- GRPO), a reinforcement learning method specifically for reasoning segmentation that learns to determine optimal segmentation timing based accumulated rea- soning context. Finally, we construct a pathology-specific reasoning segmentation benchmark of 11,6677 triplets of pathology image, ground-truth mask, query, and reasoning chain including both public and private pathology images. Zero-shot evaluation on pathology images with oft-oi-domain morphologies/pptholooggo shows 61% % improvement over state-of- th-the--rt art segmentation models. More details: http:////ggaccacccccoooo///3/5/0
INTRODUCTION
Segmentation on pathology images enables quantitative tissue analysis to support clinical diagno- sis (Zhu et al., 2025b Wang al. [2019] Zhou al., (2024b). However, clinical practice routinely encounters rare pathologies or new morphologies absent from training sets (Schäfer et al.) 2024 Zhou (ct al. al. 2024a), where closed-set segmentation methods fail to generalize. When pathologists examine uncommon tumor pathologiesmmorphooggie they typically apply domain knowledge and clinical terminology to identify malignant regions, which is a reasoning process that existing closed-ser set segmentation models cannot also replicate. Although SAM (Kirillov et al.) 2023 He et al. 2025) and variants offer open-set segmentation capabilities through prompting (e.g.. bounding boxes or point coordinates), they require intensive-manus clicking for every target instance. Moreover, they cannot interpret semantic or clinical queries expressed in text. Finally, beyond generalization to new pathologies/morpholgge
clinical pathology workflows also require explainable segmentation mod- els. which can articulate diagnostic reasoning rather than making predictions solely (van Veldhuizen
al. 2025 Wu et al. 2023).
Reasoning segmentation offers a promising pathway for open-set segmentation in pathology im- ages. Specifically, reasoning segmentation takes queries encoded in natural language to guide the segmentation on unseen objects via semantic understanding and reasoning (Lai et al. 2024). More- over, reasoning segmentation models may generate explicit reasoning chains, such that pathologists can validate against clinical knowledge (Moor al. 2023 Lin et al. 2025). Formally, Table
"Equal contribution.
1 "oorrsponniig authors: kgy11114@@rj..omma.m.m lichizhang@sjuu.ddu

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of segmentation methods for pathology. Abbreviations: "Unseen Gen.":
Generalization to unseen morphologiexxobjeets;; "Path. Spec.": patholog-sspciftte models; "Reason.". Capability for general visual reasoning and language understanding: "Stain Rob." for stain-variation- invariant representations; "RI-Seg" for segmentation-speccfe reinforcement learning.
Method
Unseen Gen.
Path. Spec.
Reason.
Stain Rob.
RL-Seg
Interaction
nnU-Net [[sensee et al. 2021
X
X
X
X
X
MedSAM Maa et al. 2024
X
X
X
X
Point/Box
SAM-Path Zhang el al. 2023
V
X
X
X
Point/Box
SegAnyPath Wang
20240
/
X
V
X
Point/Box
BiomedParse (Zhao al. 2024
X
X
X
X
Text
Seg-Zero (Liu
20255
X
V
X
X
Text
LISA (Lai et al. 2024]
X
V
X
X
Text
MMR (Jang al 2025
V
X
Text
PathChal-Segki (Ours)
V
V
Text
compares reasoning segmentation methods with existing closed-set segmentation methods for pathol- ogy analysis. However, existing reasoning segmentation models rely on general-domain vision encoder that lack specialized understanding of pathological features, as well as the robustness to stain variations (Wang et al. 2024b Ke al. 2025). Consequently, these models exhibit suboptimal generalization ability when applied to new pathologiesmmorpoologiss Moreover, LISA's 's paradigm of inserting <SEG> tokens at predetermined positions prevents autonomous determination of optimal segmentation timing. Additionally, although reinforcement learning methods such as GRPO (Shao) ct al, 2024) can train reasoning segmentation models to generate reasoning chains and segmentation accuracy concurrently (Liu et al., (025b),, they fail to identify when the reasoning segmentation model has accumulated sufficient semantic context to trigger segmentation output. This also results in suboptimal performance in pathology segmentation, which requires
complex reasoning. Finally, there is no existing reasoning segmentation dataset or benchmarks for pathology.
To address these limitations, introduce PathChat-SegR1, vathology-speccio reasoning segmenta- tion model. Specifically, PathChat-SegRI builds upon the pathology-specific vision encoder with novel reinforcement learning designed for segmentation. We reformulate <SEG> token emission from fixed insertion to autonomous generation. To further improve performance, we also train the visual encoder in PathChat-Segih S with invariatarriant ceff-distillation handle stain variations.
In brief, the major contributions of this paper are three-fold. First, we propose PathChat-SegRI, reasoning segmentation model for pathology with a sathology-speciffee vision encoder fine-tuned with novel stain- -tvininnariiant self- seff-ditillation. Second, we propose Begmentation-Optiimzze GRPO (SO-GRPO), which extends standard GRPO specifically for reasoning segmentation by identifying the optimal <SEG> generation timing during LLM reasoning. Specifically, we introduce a differ- entiable
reward for controlling redundant generation, together with adaptive scheduling for ensuring training stability. Finally, we construct a large-scale PathChat-SegRI benchmark comprising 118,667 triplets of pathology image, ground- truth mask, query, and reasoning chain, spanning Correspondingly, we propose a novel semi-automated pipeline to annotate the reasoning chain.
perrrttion---rooomannceeee reward, and sparsity-
both public and private pathology images.
2 RELATED WORKS
Pathological Image Segmentation Segmentation on pathology images helps quantify tissue mor- phology to support clinical diagnosis and treatment planning. Traditional closed-set segmentation approaches leverage general medical segmentation models (e.g., nnU-Net, MedSAM, BiomedParse) trained with pre-defined categories of pathology and limited morphology types. Yet, these methods lack pathology-specine understanding and therefore fail to generalize to new tumor-pathologies and morphologies absent from training distributions. SAM-based adaptations attempt to address generalization via box or point prompting (Kirillov et al. 2023 Shen et al.] 2025a Shi et al. (025).. For example, SAM-Path (Zhang al., 2023) adapts SAM (Kirillov et al. 2023) via trainable class embeddings to represent different morphologies, but requires extensive data set-specific fine-tuning for each pathology/morphology type, similarly to SegAnyPan (Wang et al.) 2024c Shen et al.
2

---

Published as a conference paper at ICLR 2026
2025d). However, prompting in SAM variants demands intensive manual efforts, rendering them impractical for whole slide images containing hundreds of cellular structures.
In summary, existing pathology segmentation methods achieve strong performance in training distributions under closed-set assumptions: or open-set capabilities at the cost of heavy manual prompting efforts, but not both simultaneously (Shen et al. 2025a Shi et al. 2025)
Reasoning Segmentation Reasoning segmentation generates pixel masks from text queries to avoid time-consuming point/box prompting (T.ai et al., 2024] [Shen Zhang 2025 Shen et al., 2025b). There are three major paradigms for building a reasoning segmentation model. The first paradigm employs multimodal LLMs (MLLMs) to generate prompts, i.e., points or bounding boxes, to guide a separate frozen segmentation model For example, SAM4MLLM and Seg-Zero (Chen al.) 2024 Liu et al., 2025b) generates coordinates from MLLM's ' reasoning to prompt SAM (Kirillov et al. 2023) for subsequent segmentation.
However, these types of methods suffer from performance limitations and remain sensitive to prompts as the point prompting inadequately captures the complex spatial information in pathology images. The second paradigm integrates the segmentation capability directly into LLM by connecting the segmentation decoder LLM with special tokens. For example, LISA (Lai al. (024) introduces the <SEG> token whose hidden representation feeds into a mask decoder to guide the mask segmentation, while MMR (Jang et al.) 2025) extends this approach to enable multi-gres granularity reasoning for hierarchical tissue structures. The third paradigm reformulates segmentation as text generation (Shen ct al., 2025c), with Text4Seg++ (Lan et al.) 2025) labeling image patches using textual categories to enable a direct text-to-mask pathway. Despite these advances, existing reasoning segmentation models rely on a general-domain vision encoder that lacks specialized understanding of pathology features and robustness to staining variations, resulting in limited performance.
Reinforcement Learning for Segmentation Reinforcement learning formulates segmentation as sequential decision-makinn to improve model performance Existing methods intervene at different stages, including pixel-level approaches (Liu et al.) 2025a Araslanov et al. 2019) treat each pixel as an agent with actor-critic policies but remain computationally costly for whole slide images, while auxiliary localization methods (Yi et al. 2023) train boundary-seeking controllers under PPO to reduce annotation requirements. Recent work focuses on prompting and reasoning optimization, where agents learn to produce informative prompts for foundation models. For example, Align- SAM (Huang et al., 2024) applies PPO for woint-prompt selection, while Seg-Zero, MedReasoner, and LENS (Liu al. [025bb] Yan et al. 2025 Zhu et al.) (2025a) jointly optimize chain-of-thoue thought reasoning and spatial prompts via GRPO. However, standard GRPO assigns rewards uniformly across complete generation trajectories, preventing segmentation models from identifying optimal timing for segmentation generation during the reasoning process.
3 METHODS
3.1 OVERVIEW OF PATHCHAT-SEGRI
Fig. illustrates the architecture of PathChat-SegR1. Given a pathology image and text query, PathChat-SegRP produces both reasoning chain in text and a segmentation mask through its three components. First, the vision-language model (VLM) backbone leverages the RuiPath encoder to extract pathology-specific features that inform reasoning chain generation, which contains <SEG> token to initiate segmentation. Secondly, the MedSAM encoder processes the pathology image separately to capture fine-grained spatial information needed for accurate mask generation. Finally, an adapter then bridges the <SEG> token representation from the VLM to guide the mask decoder in producing the final segmentation output. Training of PathChat-Segu SeghR goes through three stages. During pre-training, the VLM learns mathology-specinn encoder undergoes stain-invariant self-distillation concurrently to handle staining variations. During supervised fine-tuning, RathChat-SegR learns to align vision and language embeddings for reasoning segmentation. To save training costs, this stage applies LoRA to the VLM, adapters to the vision encoders, and full parameeerr training to the mask decoder. During SO-GRPO reinforcement learning, SuthChat-Seghh I improves its reasoning ability by learning to determinee when a sufficient semantic context has accumulated to generate the <SEG> token in the appropriate reasoning steps.
knowledge, where the MedSAM
3

---

Published as a conference paper at ICLR 2026
Depsder
(M. ning Image
(c] Segg-Ateee
Docoder
Image Eiocdde
Repph deokk
Pegacke
Leas Laageac
Proeare
My mikmenttarricc cancer eas t'a Secped
piók
white emifast, These odh sigrlfic only beger than the wrmesding lymphecrtis. fostaen (SPG]
creer woty colosive wot- caskkr Semaally wnill hyephasyee
00000
00م00000 [SD0]]
(0) PuthChat-egR1 Franewolk
Figure 1: PathChat-SegR1 architecture overview.
3.2 STAIN-INVARIANT VISUAL PRE-TRAINING
Pathology images exhibit staining variations from different preparation protocols and laboratory practices, causing reasoning segmentation models trained on specific staining appearances fail when reasoning about tissue morphology in images with different staining styles (Shen et al. 2022 Wang et al., 2024b; Ke al.) 2025). To address this, we apply -tvariinv self-distil lation to the pre-training of MedSAM vision encoder through masked auto-encoding combined with LAB color space augmentation to learn features that remain consistent across stain variations. Specifically, for each pathology image Zi, generate two virtually re-stained views (with 2f and 2l denoting its embedding) through random channel-wil wise linear transformations in the LAB color space via RandStainNA (Ke et al., 2023; Shen et al. 2022). The MedSAM vision encoder learns to reconstruct the masked patches from pathology images via masked autoencoding (He et al., 2021) while maintaining consistent feature representations across the two staining variations. Formally, the pre-training objective LSsL of McdSAM vision encoder is formulated
LSSL
2112 + LMAE
(1)
N
where is balancing coefficent, $1,1 [Av.1, D..1] and S1,2 4, De.22 represent stain template vectors extracted from the two augmented views (Shen et al. 2022 Wang et al. (2024)), with and D denoting the mean and STD of the color distribution in LAB space. The cosine distance term weights the feature discrepancy by the magnitude of the stain variation, assigning higher penalties when the staining differences are larger. The masked auto-encoding loss LMAE enforces reconstruction of masked regions. Here, the MedSAM vision encoder serves to extract fine-graing grained spatial features needed for precise mask generation. MedSAM processes the pathology image independently from the VL.M pathway, providing complementary spatial information that the adapter later combines with the <SEG> token representation.
Meanwhile, the VLM backbone of PathChat-SegR1 leverages Qwen2.5VL BBai et al. 2025 and re- places the vision encoder with RuiPath, a foundation model pre-trained on large-scale histopathology data using DINOv2 (Oquab et al. (2024). RuiPath provides pathology-specife visual understanding, including tissue architecture, cellular lar morphology, and disease patterns absent from natural image datasets. The invariiatti self-distillation described above also applies to the RuiPath vision coder during the pre- pre-traine training of VLM. During VLM pre- pre-training, we also train the LLM on pathology image-text pairs through next-token prediction, where the L.LM learns generate textual descriptions
4

---

Published as a conference paper at ICLR 2026
(1)
(1)
PART
Lunek
Beae
321
Airght combiin acceris vctile cocltaal aldee, fooe
right sobll irmopputte cotin porondde
colling
of Boo's Anoerc
lac alayl yall achendd layp
ian Blere cogiig murted plamuuys pus Aypess trwar'
peefee hooer pellusiigg all Alpunee The orga shod our
ratilly tur're brficinng soer any with Ace Accaplari
momer chemmatt prmperes whher mnd li
by cabidiy sscie
Figure 2: PathChat-SegRl benchmark statistics and examples. (a) Anatomical distribution showing coverage, with both public and proprietary sources mapped to their corresponding anatomical origins. (b) Magnification-levl distribution. (c) Dataset composition. (d) Representative intraoperative frozen section examples with associated reasoning chains.
conditioned on visual features from RuiPath. This stage freezes the LLM backbone while training the RuiPath encoder and intermediate projection layer to establish vision-languae alignment.
3.3 SUPERVISED
After pre-training, supervised fine-tuning teaches PathChat-SegRi to generate reasoning chains containing <SEG> token while establishing the alignment between language-base reasoning from the VLM and pixel-wise mask generation from the segmentation decoder. To preserve pre-trained knowledge from both pathology and language domains, we adopt parameter-efficient fine-tuning strategies. For the L.M backhone, and RuiPath encoder in VLM, we apply LoRA (Hu al.) (021) to the self-attention blocks. For the MedSAM encoder, we insert lightweight adapters after the multi-head attention layer and parallel to the MLP layer in each ViT block (Chen et al. 2022).
which allow MedS AM encoder to adjust its spatial features. On the contrary, the mask decoder and the adapter that bridges the <SEG> token to the decoder undergo full parameter training, because these components are initialized randomly. The supervised fine-tuning objective jointly optimizes reasoning chain generation and mask prediction, depicted as:
LSFT >cur LCE + Aseg (LiiEe + LUCE)
where the first term represents an auto-regressive cross-entropy loss for reasoning chain generation. The second term combines the Dice loss Liic and the binary cross-entropy loss LBCE for the segmentation mask. Parameters Acor and Aseg balance these two terms.
3.4 REINFORCEMENT LEARNING WITH SO-GRPO
Standard GRPO assigns rewards uniformly, preventing the VLM in reasoning segmentation from identifying when sufficient semantic context has accumulated trigger segmentation. We propose Segmentation-Opiin Optimmzzed GRPO (SO-GRPO), which extends GRPO to address three challenges: (I) determining optimal <SEG> token generation timing, (2) enabling gradient flow from segmentation quality to reasoning representations, and (3) controlling generation sparsity.
<SEG> token assignment via GAE. To identify when the VLM has accumulated sufficient seman- tic context for segmentation, we replace trajectory-level ad advantage estimation with Generalized Ad- vantage Estimation (GAE) (Schulman et al. 2018). Formally, assigns credit to individual time steps during reasoning generation, formulated as AAEE A az) = > ( where 8; represents the reasoning state at time step / (comprising the text query, visual features, and all previously generated reasoning tokens), a denotes the token generation action at that step 6, = rı + +V+++++)) - V(ss) is the temporal difference (TD) residual that measures the reward improvement between consecutive reasoning states. The state value function (sx) estimates the expected cumulative reward from state 8, onward, quantifying how much the current reasoning context contributes to the final segmentation
5

---

Published as a conference paper at ICLR 2026
quality. The discount factor " determines how much future rewards influence current decisions, while controls the bias-variance trade-off in advantage estimation. By computing advantages at each reasoning step through exponentially weighted sums of temporal difference residuals, GAE evaluates whether generating the <SEG> token at the current step would yield better segmenta- tion outcomes than continuing reasoning. The VLM learns generate the <SEG> token when A (st, dz = <SEG>) is maximized, indicating that the accumulated reasoning context contains sufficient semantic information and that further text generation would not improve segmentation quality. Moreover, GAE reduces the gradient variance by factor of 1-(>))² I-(aA]= compared to Monte
Carlo estimation, enabling more stable policy updates at the decisive generation step where the model must decide whether to trigger segmentation continue reasoning.
Differentiable segmentation performance reward. Standard segmentation metrics such as Dice and IoU are discrete and non-differentiable. Therefore, using these metrics as performance reward prevents gradient flow from to the <SEG> token representation. We introduce differentiable approx-
imations through probability softening as the performance reward, namely Rooft Σ. P.+2 gi+e'
2 E, Pe
where p. = o (Mpeed..) are softened probabilities from the predicted segmentation mask, 9i are ground truth, and < 10-7 ensures numerical stability. The softening operation converts discrete masks into continuous probability distributions, enabling a differentiable path Rooft Mpreed + /SEG where gradients flow from segmentation performance directly to the <SEG> token representation. During &Raa guides hSEG to encode spatial locations and semantic properties that maximize overlap with ground truth masks.
optimization, the gradient öhane
Sparsity-aware reward. Reasoning segmentation model may generate <SEG> tokens excessively or prematurely, degrading both computational efficiency and segmentation performance. We formulate this as an information problem, introducing sparsity aware reward that encourages generation only when the reasoning chain provides sufficient segmentation-relevan information:
Repurse
Psparse I[(s E Sspatial) Yspurse I[s² & Ssputial):
where Ssparial denotes states containing spatial-semantic information detected via rule-based patterns (e.g., presence of location descriptors such as "boundary" or "region")), I[()) represents the indicator function, and &sparse and Ysparse are balancing coefficients. This reward formulation penalizes <SEG> generation at non-spatial states by -Issuuse while encouraging at spatial states by + +Sppase- Math- ematically, this is equivalent to maximizing mutual information ((M at <SEG>> (s:) between token generation and segmentation targets.
Overall reward formulation. Finally, the overall reward function comprises segmentation quality (Rwn),, sparsity control (Rugarse), spatial grounding (Rspañat), format correctness (/rormat), and con- ciseness penalty (Rien). The spatial reward Rspatial verifies whether the predicted mask centroid aligns with location keywords extracted from the text query following previous work (Liu et al. 2025b). The format reward enforces structured reasoning with proper special token usage like <SEG> (DeepSekk] Al et al. (025).. The conciseness penalty discourages overly long chain-of thain-of-thought sequences via Rien Alen max(0, In(L. Lo + 1)), where L. denotes reasoning length and Lo is grace threshold. The overall reward in SO-GRPO can be formulated as
Jso-GRPO Eyona
Σ ACAF ar) log To (arls))
Asoii Roso + Asparse Ropaase
+ Aiputial Regutiil + Atommal
Riounat
Rien -XKL LKL
where LXL prevents excessive policy deviation from the supervised initialization, and Asots Asparse Xspatial, Aroemats ÄKL the balancing coefficients.
Convergence guarantees through adaptive scheduling. To ensure stable optimization during PathChat-Seglh reinforcement learning, employ learning rate schedules a₁ an/(1 + nk) satisfying Robbins-Moon Monro conditions Ee 0g = oo and En aj < cc. Combined with KL regulariza- tion that constrains policy updates via Lyapunov function V(0)) Ex-p [I((==|[TT) establish

---

Published as a conference paper at ICLR 2026
monotonic improvement V(0++1) V(0k) VV This inequality guaranties that each policy update decreases the divergence from the optimal policy, with the magnitude of the descent controlled by the gradient norm [VV ensuring convergence to local optimum.
3.5 PATHCHAT- BENCHMARK
Data Statistics. We introduce the PathChat-SegR1 benchmark comprising 118,667 triplets of pathology image, ground-truth mask, query, and reasoning chain spanning diverse pathology types of different morphologies, magnification levels, and clinical scenarios, as shown in Fig. 2 The benchmark integrates pathology images from six public datasets, namely Camelyon16 (Bejnordi al. 2017). Camelyon17 [Bándi et al. 2019), CRAG (Zhang] 2022), DigestPath (Da et al. 2022), GlaS (Sirinukunwattana et al. 2017). WSSS4LUAD (Han et al. 2022). covering breast, lymph node, colorectal, digestive system, and lung pathologies, accounting for 65.0% of the total dataset. To evaluate performance under authentic diagnostic conditions, we also contribute 43,847 private intraoperative frozen section pathology images from 493 real surgical procedures where patholo- gists must deliver diagnoses within minutes to guide resection decisions. These frozen sections include preparation artifacts such ice crystal distortions and tissue folding that occur during rapid cryosectioning, mimicking the challenging conditions pathologists encounter during intraoperative consultations (Wang al., 2025 2024a). The PathChat-SegR/ benchmark includes both intraop- erative microscope images and postoperative whole slide images, including rare pathologies such as chondromyxoid fibroma and chondroblastoma that are typically absent from standard training datasets. We also include a pulmonary metastasis from private bone tumors dataset containing 25,764 images with heterogeneous scanner protocols evaluate cross-domain and cross-stain-style generalization. The benchmark spans four magnification levels (5x, 10x, 20x, 40x) with 40x magni- fication comprising the largest portion to reflect clinical practice where higher resolution is needed for diagnostic assessment.
Data Annotation Process. For the public data, we reuse the ground-truth masks and provide annotation for the reasoning chain and reasoning segmentation query. For the private data, additionally first annotate the segmentation masks. Reasoning chain annotations articulate diagnostic logic rather than simple category labels, requiring downstream models explain their segmentation decisions through pathology-relevan terminology. We developed remi-automated annotation pipeline. Specifically, Gemini-2.5-PP first analyzes annotated regions to generate morphological descriptions contrasting target areas with surrounding tissue. DeepSeek-R1 (DeepSeek-Al et al. (025) then transforms these descriptions into structured reasoning chains following pathology diagnostic protocols, and three board-certified pathologists with 8-15 years of experience manually review all generated reasoning chain to verify clinical accuracy and diagnostic soundness, where revisions are made correspondingly. This pipeline ensures that reasoning annotations reflect authentic pathology diagnostic workflows while maintaining scalability for large-scale dataset construction.
4 EXPERIMENTS
Table Performance comparison across in-domain public benchmarks and private intraoperative frozen sections measured by Dice coefficient. Closed-set methods are trained separately on each dataset, while reasoning segmentation baselines and PathChat-Segee use unified training across all
eight benchmarks and tested in zero-shot manner. FS-Mic: Frozen Sections (Microscope); FS- :SI
Frozen Sections (Whole Slide Image).
Category
Methods
Cam16
Cam17
GlaS
Digest
CRAG
WSSS
FS-Mic
FS-WSI
MedSAM
et
2024
0.69
0.38
0.76
Closed-Set
SAM-Puth Zhang il 2025
0.64
0.67
0.78
0.55
0.77
0.72
0.54
0.73
neU-Net [ensee el al 2121
0.74
0.79
0.91
0.67
0.94
0.84
0.69
0.82
BiomedParse (hha
0.61
0.64
0.83
0.47
0.51
0.12
0.53
0.71
LISA-7B Lai ct
2024
0.48
0.45
0.52
0.46
0.60
0.47
0.39
0.42
OVSeg lang ci ml
J025
0.41
0.39
0.43
0.38
0.49
0.39
0.31
0.34
Reasoning Seg.
SAM4M
2024
0.47
0.62
0.57
0.64
0.52
0.51
0.56
Seg Zero
[
ar.
20225
0.54
0.55
0.66
0.52
0.59
0.38
0.42
0,51
MMR-7B (ang al.
2025
0.57
0.59
0.72
0.58
0.74
0.68
0.56
0.62
Ours
0.76
0.78
0.87
0.74
0.92
0.78
0.74
0.84

---

Published as a conference paper at ICLR 2026
Implementation Details. All experiments on 8 NVIDIA H800 GPUs using PyTorch 2.1 with AdamW optimizer (learning rate x 10 and cosine annealing schedule. The training process uses batch size 64 with gradient accumulation over 2 steps. LoRA adapters apply rank 16 with dropout probability 0.1 to the attention layers. The MedS AM encoder operates with patch size 16 and applies 75% masking ratio during stain-invariant self-distillation pretraining. SO-GRPO training balances multiple reward components with Asott = 0.3, Asparse = 0.2, Aspatial Afoemar 0.1, and KL divergence penalty λKL. 0.01 to maintain policy stability throughout reinforcement learning. We use an 8:1:1 split between training. validation, and testing sets on the proposed PathChat-Sepe 1. Zero-shot generalization is evaluated in 39,611 unseen pathological images spanning two scenarios, namely the Pulmonary Metastasis of Bone Tumors (PMBT) dataset with 25,764 images, and Rare osteosarcoma
Disease (RD) with 13,847 images of atypical pathologies such as chondroblastoma- and chondromyxoid fibroma that are absent from the training distribution. We further assess in one-shot manner by providing a single annotated reference during inference on the RD dataset.
Theskey dkggi SoT
demeee dee the onn jood triple
saatter and Abo dat
broin fate codfelionall
Lamere
de iabrogh dur
Figure Qualitative comparison of segmentation performance on challenging pathologies in one-shot in-context learning manner. Right panel shows training-free one-shot adaptation on chondromyxoid fibroma, where PathChat-SegR1 analyzes given image and annotation to extract morphological features and applies them to segment calcification in an unlabeled image with different staining appearance.
In-Domain Performance Evaluation. Tab. 2 compares PathChat-SegR1 against closed-set and reasoning segmentation methods across eight benchmarks spanning public datasets and clinical frozen sections. PathChat-SegR: achieves 0.76 Dice on Camelyon16, outperforming the strongest reasoning baseline MMR-7B by 33% relative improvement. In gland segmentation tasks, PathChat-SegR reaches 0.87 Dice on GlaS and 0.92 Dice on CRAG in a zero-shot manner, approaching dataset- specific tuned nnU-Net performance while maintaining unified training across all benchmarks. These results demonstrate that pathology-specifee visual encoding combined with reasoning capabilities enables strong reasoning segmentation performance of PathChat-SegR across diverse tissue types and pathology imaging conditions without requiring task-specific adaptation.
Table 3: Out-of-domain evaluation on unseen pathologies measured by Dice coefficient. PMBT: Pulmonary Metastasis from Bone Tumors; RD: Rare Diseases; RDw/E: Rare Diseases with one-shot example provided during inference.
Methods
Zero-shot PMBT
One-shot RDw/E
RD
LISA-7B (Lai et al. 2024
0.30
0.24
0.37
OVSeg [Lingg el al. 2023
0.29
0.25
0.36
SAM4MELM (Chen et al.
2024
0.37
0.28
0.41
Seg-Zero (Liu et al. 2025b
0.39
0.29
0.44
MMR-7B (Jang et al., 2025
0.36
0.33
0.47
PathChat-SegKI (Ours)
0.58
0.53
0.72
Out-of-Domain Evaluation on Unseen Pathologies.
Tab. evaluuaee performance in completely On the PMBT dataset containing pulmonary
unseen domains under zero-shot and one-shot conditions.
8

---

Published as a conference paper at ICLR 2026
metastases from bone tumors with heterogeneous staining protocols and scanner variations, PathChat- SegR1 achieves 0.58 Dice under zero-shot evaluation, representing 61% relative improvement over MMR-7B (0.36 Dice). On the RD collection comprising atypical pathologies absent from the training data, including chondroblastoma-lik osteosarcoma and chondromyxoid fibroma, PathChat-SegR achieves 0.53 Dice compared to MMR-7BB 0.33 Dice. In one-shot manner in which single annotated reference is provided during inference, PathChat-SegR reaches 0.72 Dice in the RD dataset, showing 53% improvement over MMR-7B (0.47 Dice). These results indicate that PathChat-Segg can be applied to clinical settings where new pathologiesmmophoooggee are encountered.
One-Shot In-Context Learning for Novel Pathologies. PathChat-Segh enables training- free adaptation through in-context visual learning, where single annotated reference image provides morphological context for segmenting novel pathologies. Fig. demonstrates this capability on a chondromyxoid fibroma case, where the PathChat- SeeCChhtt-tgRiIi receives a teaching pathology image with annotated regions, osteoclast-like giant cells, and chondroblast-like cells. When presented with an unlabeled image exhibiting different staining style, PathChat-SegR1 I can identify calcification based on structural features rather than color matching. This one-shot prompting improves RD collection performance from 0.53 Dice under zero-shot conditions to 0.72 Dice, surpassing MMR-7B (0.47 Dice) by 53%. The one-shot prompting capabilities can address the annotation burden inherent to rare pathologies.
Table 4: Component ablation study on the PMBT dataset measured by Dice coefficient. Architecture ablations remove aathology-speciffe encoders, training ablations remove pretraining stages and supervision objectives, and reward ablations remove SO-GRPO reward components.
Ablation Type
Component
Dice Score
Full Model
0.58
w/o Ruipath Encoder Seg- Adapter
0.42
Architecture
0.52
w/o Auto-emission
0.51
MedSAM Encoder Pre-training
0.51
Training
w/o SFT
0.44
w/o RL
0.40
w/o Length Reward w/o Spatial Reward
0.56
Reward Function
0.53
Ablation on Model Architecture and Training Components. Tab. 4 evaluates component contri- butions on the PMBT dataset. Removing the RuiPath encoder reduces performance by 0.16 Dice, removing reinforcement learning by 0.18 Dice, and removing SFT by 0.14 Dice. The RuiPath en- knowledge
coder degradation validates that needed recognize tissue morphologies in out-of-domain samples where morphology differs from natural image distributions. Removing MedSAM encoder pre-training with self distillation reduces performance to 0.51, confirming that itain-ivvarian& deff-disillation addresses stain variations. The reinforcement learning ablation demonstrates that SFT alone cannot optimize segmentation token timing, as SFT provides uniform supervision across all reasoning steps rather than learning when semantic context suffices for mask generation.
vision encoders lack pathology-specing
Ablation on SO-GRPO Components. Tab. 5 analyzes SO-GRPO improvements over standard GRPO on the PMBT dataset. Standard GRPO improves performance from 0.40 Dice (SFT only) to 0.53 Dice, while SO-GRPO reaches 0.58 Dice with faster convergence (18K vs 24K training steps). Removing GAE reduces performance to 0.55, confirming that trajectory-level advantage estimation in standard GRPO cannot identify optimal segmentation token timing, whereas GAE assigns credit to individual reasoning steps. Removing differentiable rewards reduces performance to 0.56, discrete Dice metrics block gradient flow from segmentation quality to token representations.
Ablation on Reasoning Quality. Tab. 6 evaluates reasonign chain generation quality FS-WSI and PMBT datasets measured by BLEU-4 and Fl scores. PathChat-Seghh achieves 0.315 BLEU-4 and 0.612 F1 FS-WSI, exceeding reasoning segmentation baselines LISA (0.281, 0.568) and

---

Published as a conference paper at ICLR 2026
Table 5: SO-GRPO component ablation the PMBT dataset. Performance measured by Dice coefficient, convergence measured by training steps to reach optimal performance, and training stability measured by gradient variance during policy updates.
Configuration
Dice
Steps
Grad. Var.
Full SO-GRPO
0.58
18K
0.031
Baseline Methods
SFT only
0.40
Standard GRPO
0.53
24K
0.048
SO-GRPO Ablations
w/o GAE
0.55
22K
0.042
w/o Differentiable reward
0.56
20K
0.038
w/o Sparsity-awan reward
0.57
19K
0,035
w/o Adaptive scheduling
0.57
21K
0.033
Seg-Zero (0.279, 0.571), as well as medical VLMs LLaVA-Med (0.285, 0.579) and Med-PaLM (0.291, 0.581). The improvement over Med-PaLM, which incorporates medical text pretraining, indicates that catthloggyspeccffie visual knowledge from RuiPath contributes reasoning quality beyond language model capabilities. These results validate that SO-GRPO optimization improves both segmentation accuracy and reasoning interpretabilityy addressing the requirement for models to articulate diagnostic criteria rather than produce opaque predictions.
Table 6: Reasoning chain quality evaluation on intraoperative frozen sections and rare disease datasets. Quality measured by BLEU-4 for n-gram overlap and FI for semantic alignment with expert-annotated reasoning chains. FS-WSI: Frozen Sections (Whole Slide Image); PMBT: Pulmonary Metastasis
from Bone Tumors.
FS-WSI
PMBT
Model
BLEU-4
F1
BLEU-4
F1
LISA (Lai et al. 2024
0.281
0.568
0.275
0.588
Seg Seg-Zero (Liu et al. 2025b)
0.279
0.571
0.266
0.575
MMR (Jang al. 202
0.272
0.562
0.278
0.581
Med
0.285
0.579
0,288
0.595
Mcd-PaLM
0.291
0.581
0.285
0.589
Path Chatt-egRRR (Ours)
0.315
0.612
0.311
0.607
CONCLUSION
We present PathC comprehensive framework addressing the convergent challenges of reasoning segmentation in clinical pathology. Our contributions span three dimensions. First, we develop the PathChat-SegR1 Framework integrating pathology-spece visual encoders with tonomous <SEG> token generation, departing from fixed injection paradigms to enable context-aware segmentation timing. Second, introduce SO-GRPO, extending standard GRPO through temporal credit assignment, differentiable reward approximation, sparsity control, and convergence guaran- tees, providing a principled optimization approach for special token generation tasks, evaluation protocols for the field. Third, we construct the PathChat-SegRP Benchmark comprising 118,667 image-mask- meee-mmass--rrinnggggg triplets including challenging intraoperative frozen sections. Our experimental results demonstrate the framework's robustness in complex clinical scenarios. On intraoperative frozen sections with severe artifacts, the model maintains 0. 0.74-0.84 Dice scores. zero-shot eval- uation on rare diseases, it achieves 0.53 Dice, with one-shot adaptation improving to 0.72 without retraining. These results suggest that integrating domain-specific knowledge, reinforcement learning optimization, and structured reasoning image segmentation. The framework' ability to adapt to unseen pathologics through single-example prompting offers practical value for clinical deployment where annotated data remains scarce.
enhance both generalization and interpretability in medical
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62471288).
REFERENCES
Nikita Araslanov, Constantin A. Rothkopf, and Stefan Roth. Actor-critic instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8758-8767, 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2.5-vl technical report. arXiv, 2025.
Péter Bándi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 challenge. IEEE Transactions on Medical Imaging, 38(2)550--560, 2019. doi: 10.1 201.TM1..018.286777777
Babak Ehteshami Bejnordi, Mitko Veta, Nico Karssemeijer, Geert Litjens, Meyke Hermsen, Quirine Manson, Maschenka Balkenhol, Oscar Geessink, Nikolaos Stathonikos, Peter Bult, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA Neurology, 318(22):2199 2210, 2017. doi: 10.100//aam..217.111
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-1661) 2022.
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi-modal large language model for referring expression segmentation. arXiv, 2024.
Qian Da, Xiaodi Huang, Zhongyu Li, Yanfei Zuo, Chenbin Zhang, Jingxin Liu, Wen Chen, Jiahui Li, Dou Xu, Zhiqiang Hu, al. Digestpath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Medical Image Analysis, 80: 80:102485, 2022. doi: 10.1016/j.media. 2022.
111016f..ediaa2222.10000000 102485.
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv, 2025.
Chu Han, Xipeng Pan, Lixu Yan, Huan Lin, Bingbing Li, Su Yao, Shanshan Lv, Zhenwei Shi, Jinhai Mai, Jiatai Lin, et al. Wsss4luad: Grand challenge weakly-supervised tissue semantic segmentation for lung adenocarcinoma, arXiv, 2022.
Dexuan He, Xiao Zhou, Wenbin Guan, Liyuan Zhang, Xiaoman Zhang, Sinuo Xu, Ge Wang, Lifeng Wang, Xiaojun Yuan, Xin Sun, et al. Boosting pathology foundation models via few-shot prompt-tuning for rare cancer subtyping. arXiv preprint arXiv:2508. 25904 2025.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv, 2021.
Duojun Huang, Xinyu Xiong, Jie Ma, Jichang Li, Zequn Jie, Lin Ma, and Guanbin Li. Alignsam: Aligning segment anything model to open context via reinforcement learning. arXiv preprint arXiv:2406. 00480, 2024.
Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnu-net: self-configuring method for deep learning- based biomedical image segmentation. Nature Methods, 18(2):203-111, 2021.
11

---

Published as a conference paper at ICLR 2026
Donggon Jang, Yucheol Cho, Suin Lee, Tachyeon Kim, and Dae-Shik Kim. Mmr: A large- benchmark dataset for -ulli-target and multi- gultiigranullrity reasoning segmentation. arXiv preprint arXiv:2503.13881, 2025.
Jing Ke, Kai Liu, Yuxiang Sun, Yuying Xue, Jiaxuan Huang, Yizhou Lu, Jun Dai, Yaobing Chen, Xiaodan Han, Yiqing Shen, and Dinggang Shen. Artifact detection and restoration in histology images with stain-style and structural preservation. IEEE Transactions on Medical Imaging, 42
(12):3487-3500, 2023.
doi: 10.1 1 2099TTM1.0223328888
Jing Ke, Yijin Zhou, Yiqing Shen, Yi Guo, Ning Liu, Xiaodan Han, and Dinggang Shen. Learnable color space conversion and fusion for stain normalization in pathology images. Medical Image Analysis, 101:103424, 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo. Piotr Dollár, and Ross Girshick. Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3897-3908, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv, 2024.
Mengcheng Lan, Chaofeng Chen, Jiaxing Xu, Zong- Zong-Han Li, Yiping Ke, Xudong Jiang, Yingchen Yu, Yunqing Zhao, and Song Bai. Text4seg++: Advancing image segmentation via generative language modeling. arXiv preprini vrXiv:2509.0632/ 2025.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7061-7070, 2023.
Haoneng Lin, Cheng Xu, and Jing Qin. Taming vision-language models models for medical image analysis: A comprehensive review, 2025. UKL/A/pppe//aarxxrrrrr////////// 18378
Yunxin Liu, Di Yuan, Zhenghua Xu, Yuefu Zhan, Hongwei Zhang, Jun Lu, and Thomas Lukasiewiez. Pixel level deep reinforcement learning for accurate and robust medical image segmentation. Scientific Reports, 15(1): 2025a.
Yuqi Liu, Bohao Peng, Zhisheng Zhong. Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chaie guided segmentation via cognitive reinforcement. arXiv, 2025b.
Jun Ma. Yuting He, Feifci Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15:654, 2024.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasumaga Cyril Zakka, Yash Dalmia, Ed- uardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: multimodal medical few-shot learner, 2023. URL
wwt.a///rrga////oaaaaaoaa..sssss 2307 .5188
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv, 2024.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URJattpa://rrxiv.orr
abs 0003/156... 02438
Raphael Schäfer, Till Nicke, Henning Höfener, Annkristin Lange, Dorit Merhof, Friedrich Feuerhake, Volkmar Schulz, Johannes Lotz, and Fabian Kiessling. Overcoming data scarcity in biomedical imaging with foundational multi-task model. Nature Computational Science, 4(7):495-500 07 2024. ISSN 2662-8457.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
12

---

Published as a conference paper at ICLR 2026
Yiqing Shen and Dell Zhang. A survey of language-guide video object segmentation: from referring to reasoning. Vicinagearth, 2(1):1-20, 2025.
Yiging Shen, Yulin Luo, Dinggang Shen, and Jing Ke. Randstainna: Learning stain-agnostic features from histology slides by bridging stain augmentation and normalization. International Conference Medical Image Computing and Computer-Asistted Intervention, pp. 212-22122 2211. Springer, 2022.
Yiqing Shen, David Dreizin, Blanca Inigo, and Mathias Unberath. Protosam-2d: 2d semantic segment anything model with mask-level prototype-leaming and distillation. Medical Imaging 2025: Image Processing, volume 13406, pp. 18-22. SPIE, 2025a.
Yiqing Shen, Chenjia Li, Fei Xiong, Jeong-O Jeong, Tianpeng Wang, Michael Latman, and Mathias Unberath. Reasoning segmentation for images and videos: A survey. arXiv preprint arXiv:2505.18816, 2025b.
Yiqing Shen, Bohan Liu, Chenjia Li, Lalithkumar Seenivasan, and Mathias Unberath. Online reasoning video segmentation with just-in-time digital twins. arXiv preprint arXiv:2503.22 1056, 2025c.
Yiqing Shen, Chong Wang, and Jing Ke. Autopathml: Automated machine learning for histology images via large language model and multi-agent. Artificial Intelligence for Engineering, 1(1): 32 43, 2025d.
Wentao Shi, Junjun He, and Yiqing Shen. Sit-sam: A semantie-integration transformer that adapts the segment anything model zero-shot medical image semantic segmentation. Biomedical Signal Processing and Control, 110: 108086, 2025.
Korsuk Sirinukunwattana. Josien P.W. Pluim, Hao Chen, Xiaojuan Qi, Pheng-Ant Heng, Yun Bo Guo, Li Yang Wang, Bogdan J. Matuszewski, Elia Bruni, Urko Sanchez, et al. Gland segmentation in colon histology images: The glas challenge contest. Medical Image Analysis, 35:489-5555 502, 2017. doi: 10. 10.1000.mmmii.2200 m0d.i..201e21... 2016.08.000
Vivien van Veldhuizen, Vanessa Botha, Chunyao Lu, Melis Erdal Cesur, Kevin Groot Lipman, Edwin D. de Jong, Hugo Horlings, Clárisa I. Sanchez, Cees G. M. Snoek, Lodewyk Wessels, Ritse Mann, Eric Mareus, and Jonas Teuwen. Foundation models in medical imaging review and outlook, 2025. URL https: wttpiviivvv.m///..//.a.//aaaaava org/ hatg//bb//// abs /20../506.... 09095
Chong Wang, Zhenqi He, Junjun He, Jin Ye, and Yiging Shen. Histology image artifact restoration with lightweight transformer based diffusion model. In International Conference on Artificial Intelligence in Medicine, pp. 81-89. Springer, 2024a.
Chong Wang, Shuxin Li, Jing Ke, Chen Zhang, and Yiqing Shen. Randstainna++: Enhance random stain augmentation and normalization through foreground and background differentiation. IEEE Journal of Biomedical and Health Informatics, 28(6):36603360 3671, 2024b.
Chong Wang. Yajie Wan, Shuxin Li, Kaili Qu, Xuezhi Zhou, Junjun He, Jing Ke, Yi Yu, Tianyun Wang, and Yiqing Shen. Seganypath: A foundation model for multi-resolution stain-variant and multi-task pathology image segmentation. IEEE Transactions on Medical Imaging, 2024c. doi: 10.1 N0.1009/TMI.00 2000/TTT2.222455133322
Chong Wang. Kaili Qu, Shuxin Li, Yi Yu, Junjun He, Chen Zhang, and Yiqing Shen. Artidiffuser: A unified framework for artifact restoration and synthesis for histology images via counterfactual diffusion model. Medical Image Analysis, 102:103567, 2025.
Shidan Wang, Donghan M. Yang, Ruichen Rong, Xiaowei Zhan, and Guanghua Xiao. Pathology image analysis using segmentation deep learning algorithms. The American Journal of Pathology, 18869)1168611698 2019. ISSN 0002-9440.
Chaoyi Wu. Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:23 10.0000000000 10.09909, 2023.
13

---

Published as a conference paper at ICLR 2026
Zhonghao Yan, Muxi Diao, Yuxuan Yang. Jiayuan Xu, Kaizhou Zhang, Ruoyan Jing, Lele Yang, Yanxi Liu, Kongming Liang, and Zhanyu Ma. Medreasoner: Reinforcement learning drives rea- soning grounding from clinical thought to pixel-level precision. arXiv preprint arXfv:25088.8817, 2025.
Weixi Yi, Vasilis Stavrinides, Zachary M.C. Baum, Qianye Yang, Dean C. Barratt, Matthew J. Clarkson, Yipeng Hu, and Shaheer U. Saeed. Boundary-rl: Reinforcement learning for weakly- supervised prostate segmentation in trus images. arXiv preprint a1X1v:2308.11376 2023.
Jingwei Zhang, Ke Ma, Saarthak Kapse, Joel Saltz, Maria Vakalopoulou, Prateek Prasanna, and Dimitris Samaras. Sam-path: A segment anything model for semantic segmentation digital pathology. arXiv, 2023.
Xiaoyu Zhang. dtaagaatasetauttoccoo, 2022.
Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lec, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, Christine Moung- Wen, et al. Biomedparse: a biomedical foundation model for image parsing everything everywhere all at once. arXiv, 2024.
Xiao Zhou, Luoyi Sun, Dexuan He, Wenbin Guan, Ruifen Wang, Lifeng Wang, Xin Sun, Kun Sun, Ya Zhang, Yanfeng Wang, et al. A knowledge-enhancee pathology vision-language foundation model for cancer diagnosis. arXiv preprint 2xXi:2242.133126, 2024a
Xiao Zhou, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge- enhanced visual-language pretraining for computational pathology. In European Conference Computer Vision, pp. 345-362. Springer. 2024b.
Lianghui Zhu, Bin Ouyang, Yuxuan Zhang, Tianheng Cheng, Rui Hu, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Li Yu, Wenyu Liu, and Xinggang Wang. Lens: Learning to segment anything with unified reinforced reasoning. arXiv preprint arXiv:2508. 14153, 2025a.
Yi Zhu, Yawen Zhu, Hongtao Ma, Bin Li, Luyao Xiao, Xiaxu Wu, and Manzhou Li. Enhancing boundary precision and long-rang range dependency modeling in medical imaging via unified attention framework, Electronics, 14(21), 2025b. ISSN 2079-9292.
14


Reviews:
[{"review_id": "2pn4pplTal", "reviewer": "Reviewer_vLfa", "summary": "The paper introduces PathChat-SegR1, a reasoning-based segmentation framework for digital pathology that integrates domain-specific visual encoders, autonomous token generation, and reinforcement learning optimization.\nThe work aims to address three main challenges in pathological image segmentation: (1) adaptation to diverse staining protocols and rare disease morphologies, (2) the need for interpretable, reasoning-driven segmentation, and (3) the absence of standardized evaluation benchmarks for such reasoning systems. The proposed framework combines four technical components: (i) a pathology-specific visual backbone, obtained by fusing RuiPath and MedSAM encoders, trained for stain invariance via masked autoencoding and self-distillation; (ii) an autonomous <SEG> token emission mechanism, which allows the model to decide when segmentation should occur during its reasoning process, replacing fixed token insertion used in prior models such as LISA; (3) a novel reinforcement-learning algorithm, SO-GRPO (Segmentation-Optimized Generalized Reward Policy Optimization), designed to improve multi-step reasoning and segmentation quality by introducing temporal credit assignment (via GAE), differentiable reward propagation, sparsity regularization, and stability guarantees through Robbins–Monro scheduling; (4) a large-scale PathChat-SegR1 Benchmark, comprising 118,667 (image, mask, reasoning) triplets collected from public pathology datasets and clinical frozen-section slides, intended to evaluate both segmentation and reasoning performance.\n\nExperimental evaluation covers zero-shot and one-shot adaptation to unseen pathologies and experiments on  public datasets (Camelyon, GlaS, DigestPath, CRAG). These experiments demonstrate competitive performance with strong supervised baselines, while maintaining explainable, reasoning-based outputs. Ablation studies and extended appendix analyses show that each component of SO-GRPO contributes to stability and accuracy.\n\nOverall, the paper presents an ambitious and comprehensive framework combining domain adaptation, language-driven reasoning, and reinforcement learning for interpretable medical segmentation, accompanied by a new benchmark dataset for this emerging research direction.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes a technically ambitious framework that combines reasoning-based segmentation, pathology-specific pretraining, and reinforcement learning optimization. Its originality lies in merging three components that have rarely been unified before:\n(1) domain-specific encoders (RuiPath + MedSAM) with stain-invariant pretraining,\n(2) autonomous generation of the special <SEG> token that determines when segmentation should occur, and\n(3) the new Segmentation-Optimized GRPO (SO-GRPO) algorithm, which adapts reinforcement learning to multi-step reasoning and non-differentiable segmentation metrics.\nThis combination is novel and directly addresses the limitations of prior models such as LISA that relied on fixed token insertion.\n\nIn terms of research quality, the experimental evidence in Table 1 is strong.\nOn intra-operative frozen sections (FS-Mic / FS-WSI), PathChat-SegR1 achieves 0.74 / 0.84 Dice, outperforming MedSAM (0.62 / 0.76) and all other reasoning-based baselines (e.g., LISA-7B = 0.42 / 0.51, MMR-7B = 0.56 / 0.62).\nOn zero-shot pulmonary metastasis segmentation (PMBT), the method reaches 0.58 Dice, over 60 % higher than previous reasoning models, and its one-shot adaptation improves this to 0.72 Dice on rare disease cases.\nThese results demonstrate good generalization to unseen tissue types and realistic clinical conditions.\nSuch robustness to staining variation and low-data regimes is an important contribution for pathology.\n\nThe SO-GRPO ablations further confirm that each algorithmic component contributes to stability and convergence, with full SO-GRPO improving Dice by +5 points over standard GRPO and reducing gradient variance by 35%.\nThe reasoning outputs are interpretable and clinically aligned: the model autonomously focuses on fibrous-capsule regions when detecting metastases, mimicking expert behavior.\nThis supports the authors’ claim of improved clinical interpretability.\n\nRegarding clarity and reproducibility, the supplementary material includes detailed pseudo-code, convergence proofs, and additional ablations, which significantly strengthen the technical completeness of the work.\nWhile the main text is dense, the appendix provides the necessary information for replication.\n\nFinally, the significance of the study is high.\nThe integration of reasoning, segmentation, and reinforcement learning defines a new research direction for reasoning-driven medical imaging.\nThe accompanying PathChat-SegR1 benchmark (118 k triplets) could become a valuable large-scale dataset for this task and can serve as a reference for future work.", "weaknesses": "Although the paper proposes an ambitious and conceptually interesting framework, several aspects limit its clarity, reproducibility, and empirical conclusiveness.\n\n1. Complex and dense exposition.\nThe paper is difficult to follow due to its highly technical writing style and extensive mathematical derivations.\nThe reader must infer how the various modules (RuiPath, MedSAM, <SEG> token, SO-GRPO) interact during pretraining, fine-tuning, and inference. A concise algorithmic diagram and explicit explanation of data flow at each stage would significantly improve readability.\n\n2.Ambiguity in benchmark construction.\nThe PathChat-SegR1 Benchmark relies partly on reasoning text generated by large language models (Gemini, DeepSeek) and later refined by human experts. The extent of human correction and the inter-annotator consistency are not quantified, raising concerns about annotation bias. While the dataset is valuable, its semi-synthetic nature should be discussed more critically.\n\n3. Limited external validation.\nThe model and benchmark are developed by the same authors.\nAlthough results on public datasets are reported later, it is unclear whether hyper-parameters were tuned using the same benchmark, which may lead to optimistic estimates. Independent validation on unseen institutional data would strengthen the clinical relevance of the claims.\n\n4. Mixed quantitative evidence (Table 2).\nOn several standard datasets, nnU-Net and other segmentation-only baselines outperform the proposed model, for example, nnU-Net achieves 0.91 Dice on GlaS and 0.94 on CRAG, compared to PathChat-SegR1’s 0.87 and 0.92. The authors argue that their model offers interpretability and generalization, yet the trade-off between reasoning quality and raw segmentation accuracy is not quantitatively assessed. A joint metric or multi-objective analysis would clarify whether the reasoning component justifies the performance gap.\n\n5. Inconsistent citation of baselines.\nTable 1 lists multiple comparison methods (MedSAM, LISA-7B, MMR-7B, etc.) without bibliographic references, while Table 2 includes them. This inconsistency makes it difficult to determine which implementations or versions were used. All baselines should be properly cited, and the paper should specify whether results come from official checkpoints or internal re-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CAYfW9rHRS", "reviewer": "Reviewer_751j", "summary": "This paper presents PathChat-SegR1, a reasoning segmentation framework for digital pathology that combines pathology-specific visual encoders, autonomous <SEG> token generation, and reinforcement learning (SO-GRPO) for improved interpretability and domain adaptation. The authors further introduce a large-scale PathChat-SegR1 Benchmark (118k image–mask–reasoning triplets) and demonstrate strong performance across zero-shot and one-shot settings, including challenging intraoperative frozen sections. The study aims to unify reasoning-driven interpretability with high-performance segmentation in clinical contexts.", "questions": "plz see my detailed comments above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths\n1. The combination of pathology-specific encoders with reinforcement-learning-optimised token emission represents an original and conceptually coherent contribution to multimodal pathology AI.\n2. The SO-GRPO algorithm is rigorously formulated, with explicit derivations of temporal credit assignment, differentiable reward propagation, and convergence guarantees—rarely seen in medical imaging work.\n3. Evaluation spans public and clinical datasets, including frozen sections and rare diseases, demonstrating robustness and strong zero-/one-shot generalisation.\n4. The reasoning-augmented segmentation outputs offer interpretable chains-of-thought that could support clinical decision-making, an important step toward trustworthy AI in pathology.\n5. The release of a large and well-annotated benchmark is a significant service to the research community.", "weaknesses": "Weaknesses \n1. The paper is dense, sometimes obscuring its main contributions. Key ideas (e.g., <SEG> token autonomy) are diluted by extended mathematical detail that could be condensed or moved to the appendix.\n2. The training protocols, data splits, and baseline implementations are only briefly described. Reproducibility would benefit from clearer procedural descriptions and access to full code upon publication.\n3. While segmentation metrics are comprehensive, there is little quantitative or user-based assessment of the quality or utility of the reasoning text for clinical interpretation.\n4. The conceptual structure resembles LISA and Seg-Zero. The incremental novelty of SO-GRPO relative to existing GRPO extensions needs clearer articulation beyond theoretical derivation.\n5. Although the appendix includes an ethics statement, more transparency about data curation, de-identification, and inter-observer variability would strengthen credibility for clinical translation.\n6. The manuscript’s prose, though technically precise, is occasionally verbose and repetitive. The narrative could be tightened for readability and to align with ICLR’s 9-page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hKg86xkzaS", "reviewer": "Reviewer_aana", "summary": "The paper introduces PathChat-SegR1, a framework that unifies a pathology-specific vision-language model (RuiPath encoder + LLM) with a segmentation decoder inspired by MedSAM. The model autonomously emits a <SEG> token to trigger segmentation reasoning and is optimized using a new reinforcement learning variant, SO-GRPO, which aims to balance reasoning quality and segmentation Dice through differentiable Dice or IoU rewards. A new benchmark, PathChat-SegR1-Bench, with over 118k reasoning–mask pairs is also proposed. Results show strong reasoning coherence and competitive segmentation performance across several datasets.", "questions": "1. Can you provide empirical evidence (variance plots, ablation) that SO-GRPO actually stabilizes training compared to GRPO or PPO?\n\n2. What is the isolated contribution of autonomous <SEG> emission versus fixed prompt insertion under matched encoders?\n\n3. How was duplication and data leakage controlled in PathChat-SegR1-Bench, particularly across magnification levels and scanners?\n\n4. How were LLM-generated reasoning chains quality-checked? Any inter-rater agreement statistics?\n\n5. How does PathChat-SegR1 compare to strong open-vocabulary segmentation models (e.g., Med-GPT4-SAM, LLaVA-Med-SAM) when trained under the same data and compute budget?\n\n6. Could the authors release training scripts, trained weights, splits, and logs to ensure full reproducibility and external validation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and organized, with clear motivation, systematic methodology, and coherent narrative flow between reasoning and segmentation.\n\n2. The integration of autonomous reasoning and segmentation via the <SEG> token is novel and intuitively powerful, showing a concrete step toward coupling cognitive reasoning with dense visual prediction.\n\n3. The experimental and benchmark contributions are substantial, covering multiple pathology modalities, providing large-scale data with reasoning supervision, and demonstrating consistent gains.", "weaknesses": "1. The SO-GRPO theoretical claims (variance reduction, convergence guarantee) depend on strong assumptions not satisfied in multimodal, non-stationary RL; empirical validation of these properties is missing.\n\n2. The differentiable Dice or IoU reward may misalign with true evaluation metrics; no calibration study verifies that reward improvements translate to better segmentation.\n\n3. The autonomous <SEG> emission is insufficiently isolated; no ablation compares it directly with fixed-token baselines to show causal benefit.\n\n4. Benchmark documentation lacks details on deduplication, inter-rater reliability, and site-level separation; LLM-generated reasoning may introduce bias or leakage.\n\n5. Reproducibility remains limited: training scripts, trained checkpoints, splits, and RL logs are not publicly available, preventing independent verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKfD5TtSfk", "reviewer": "Reviewer_t6jx", "summary": "This paper proposes PathChat-SegR1, a reasoning-based segmentation framework specifically designed for pathological image analysis. The approach integrates domain-specific vision encoders (RuiPath and MedSAM) with stain-invariant pretraining and enables autonomous emission of a special \\<SEG\\> token. To enhance reasoning quality, the authors propose SO-GRPO, a reinforcement learning algorithm adapted from GRPO to address the challenges in special-token generation: credit assignment, gradient discontinuity, generation redundancy, and convergence instability. Additionally, the authors contribute the PathChat-SegR1 Benchmark, a large-scale dataset comprising 118,667 image–mask–reasoning triplets and report strong performance among reasoning segmentation approaches in zero-shot and one-shot settings.", "questions": "Please see the \"Weaknesses\" above.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper clearly identifies the fundamental challenges in pathological image segmentation.\n2. The combination of RuiPath and MedSAM with stain-invariant pretraining to incorporate pathological priors is reasonable.\n3. The PathChat-SegR1 Benchmark is valuable to the research community.", "weaknesses": "1. Missing or inconsistent definitions: The SO-GRPO algorithm section is poorly specified. Key terms such as $R({\\tau})$ and $S_{\\text{spatial}}$ in Eq.10, $R_{\\text{soft}}$ in Eq.12, the \"Length Reward\" and \"Spatial Reward\" in Table 3 lack formal definitions. The objective function of PathChat-SegR1 is inconsistent between Eq.12 and Eq.72. Symbols $\\beta$ and $\\lambda$ are reused across different sections to denote distinct coefficients.\n2. Missing Implementation Details: Critical hyperparameters are not reported: $\\alpha$, $\\gamma$ in Eq.4; $\\lambda_{\\text{CoT}}$, $\\lambda_{\\text{seg}}$ in Eq.5; $\\lambda$ in Eq.8; and $\\beta$, $\\gamma$ in Eq.10. Algorithm 1 mentions \"value network parameters $\\phi$\" and the computation of $\\hat{A}_{\\text{GAE}}$, but there is no description of its architecture, update schedule, or training stability techniques.\n3. The use of GAE for credit assignment is not novel. Actually, introducing a value model increases training overhead, yet the paper provides no analysis of this trade-off. Besides, the variance reduction derivation in Appendix A4.4 assumes independence of TD residuals and a constant variance $\\text{Var}[\\delta_t] = \\sigma^2 $, which may not hold in practice.\n4. Soft Dice and soft IoU losses have been standard in segmentation literature for years. The claimed contribution of \"overcoming non-differentiability\" in Section 2.3.2 is unclear, as differentiable surrogates for these metrics are well-established.\n5. The sparsity reward $R_{\\text{sparse}}$ (Eq.10) depends on an indicator $I(s_t \\in S_{\\text{spatial}})$, but $S_{\\text{spatial}}$ is never defined or estimated. Appendices A.4.6–A.4.7 inconsistently switch between sparsity formulations: penalizing the policy $\\pi(\\cdot|s)$ versus penalizing entropy $H(\\pi)$. Mutual information terms are introduced without any computable estimator (e.g., MINE, NWJ) or practical surrogate, rendering the approach non-implementable.\n6. Convergence claims lack rigorous proof or supporting theoretical analysis. The use of AdamW optimizer with cosine annealing is a standard engineering choice and should not be presented as a methodological contribution.\n7. PathChat-SegR1 significantly outperforms baselines (e.g., Seg-Zero), even though Seg-Zero is also trained with RL. The paper does not adequately explain this large performance discrepancy.\n8. While the qualitative example of the One-Shot Adaptation is compelling, the paper omits quantitative and algorithmic details on how the single reference example is integrated.\n9. Reproducibility Concerns: The anonymized code repository provided by the authors is empty, potentially hindering reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)