Gate to the Vessel: Residual Experts Restore What SAM Overlooks
Authors: Weili Jiang
Jinrong Lv
Xun Gong
Xiaomeng Li
Chubin Ou
Venue: NeurIPS 2025 poster ()
Keywords: Medical Image Segmentation ; Foundation Models;  Residual Learning;  Sparse Expert Modules
Primary area: machine_learning_for_sciences

Abstract: Foundation segmentation models like Segment Anything (SAM) exhibit strong generalization on natural images but struggle with localized failures in medical imaging, especially on fine-grained structures such as vessels with complex morphology and indistinct boundaries. To address this, we propose FineSAM++, a structure-aware sparse expert framework designed to refine SAM outputs by introducing a confidence-driven soft Routing Module. This module dynamically identifies structurally uncertain regions and activates a lightweight Residual Expert to model and correct residual structural errors only within these areas, thereby achieving efficient "refinement over retraining." Extensive experiments on five public vascular segmentation datasets demonstrate that FineSAM++ consistently outperforms both SAM-adapted baselines and task-specific models in terms of accuracy, topological consistency. Our results highlight the effectiveness of sparse, structure-driven Mixture-of-Experts (MoE) strategies for enhancing the reliability of foundation vision models in clinical image understanding tasks.

Gate to the Vessel: Residual Experts Restore What SAM Overlooks
Weili Jiang School of Computing and Artificial Intelligence Southwest Jiaotong University jiangweil witeeeiiiiiiiiie edu. cn
Jinrong Lv School of Computing and Artificial Intelligence Southwest Jiaotong University 1vj ivjinrong swjtu. edu, cn
Xun Gong
School of Computing and Artificial Intelligence Southwest Jiaotong University xgongdawjet edu. cn
Xiaomeng Li
Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology eexmli suutttuuttttcmm hk
Chubin Ou"
Institute of Biomedical Enginecring, Peking University Shenzhen Graduate School Department of Radiology, Guangdong Provincial People's Hospital cou@connect. uut.hk hk
Abstract
Foundation segmentation models like Segment Anything (SAM) exhibit strong generalization on natural images but struggle with localized failures in medical imaging, especially on fine-grained structures such as vessels with complex mor- phology and indistinet boundaries. To address this, propose FineSAM++, a structure-awarc sparse expert framework designed to refine SAM outputs by in- troducing confidence-driven soft Routing Module. This module dynamically identifies structurally uncertain regions and activates a lightweight Residual Expert model and correct residual structural errors only within these areas, thereby achieving efficient "refinement over retraining." Extensive experiments on five public vascular segmentation datasets demonstrate that FineSAM++ consistently outperforms both SAM --aatteeed baselines and task-specific models in terms of accuracy, topological consistency. Our results highlight the effectiveness sparse, structure-driven Mixture-of-Exppee (MoE) strategies for enhancing the reliability of foundation vision models clinical image understanding tasks.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
SOTA SAM Variants
Fi++SAM++
Topological Errors
Figure 1: Illustration of the core motivation behind FineSAM++:
Foundation vision models such Segment Anything Model (SAM)[ have demonstrated remarkable prompt responsiveness and zero-shot generalization capabilities for natural image segmentation tasks. With the increasing popularity of SAM in general-purpose computer vision, recent efforts have explored its adaptation to medical image segmentation 42 However, empirical studies consistently report substantial performance degradation when SAM is applied to medical structures, especially for fine-grained targets like vessels, where predictions often suffer from topological errors, including disconnections, boundary ambiguity, and local omissions
5
(see [.
To bridge the domain gap between natural and medical images, existing approaches have explored domain adaptation strategies including Adapters [6 7). LoRA (8), prompt generation [9]. and SAM-CLIP hybrid models [TO ITI
(see Fig2). Nevertheless, these methods primarily focus on aligning global semantic representations, leaving structurally ambiguous uncertain regions under- modeled [ 4). As a result, they fail to resolve the persistent issuc of local structural degradation. We further observe that mispredictions in medical images predominantly occur around blurred boundaries or fine structural details, which typically manifest as high uncertainty or large residual deviations from the ground truth. This suggests that unified global adaptation scheme inherently struggles to satisfy both semantic alignment and structural recovery, as the modeling objectives present a natural conflict.
Logeod:
Frozeet
Leamable,
Adapter,
LORA
Patch Embedding
ViT Encader
VTT Encoder
VIT Encoder
Prompt Encoder
VIT Frecder
SAM Layer
SAM Layer
Mask
Gating Module
SAM Layer N
CMAtt
Exper
Exper
Mask Decoder
Ann
Mask Decoder
Ebenschicall Mask Decoder
(A)SAMed
(B) AutoSAM
(C) SAM-Adapier
(D) H-SAM
(E) Ours
Figure 2: Comparison of SAM-based adaptation methods for medical image segmentation.
Inspired by the sparse activation of Mixture-of Ex-errts (MoE) architectures in large model design 14, we argue that MoE provides a promising paradigm for fine- fine-grained structure modeling. By dynamically activating expert pathways only where necessary, an MoE-inspinn
framework can focus modeling capacity on local difficult regions without sacrificing overall efficiency. Based on this principle, we propose FineSAM++, a structure-enhanced framework following the MoE philosophy. introduces two expert modules: global LoRA Expert for domain adaptation and Residual Expert for local residual correction and structural refinement. Their activation and

---

cooperation are jointly controlled by an uncertainty eeeetaitt--waarr aware Gating Module, resulting in an efficient shared-backbone + localized refinement strategy. Our main contributions are summarized follows.
We propose FineSAM++, the structure-aware sparse MoE framework that integrates multiple localized residual correction pathways into frozen foundation segmentation models like SAM, addressing their systematic failures on fine-grained medical structures.
We design spatially-aware soft routing mechanism that jointly predicts spatial uncertainty masks and fractional expert routing weights, dynamically activating only small subset of residual experts for structurally ambiguous regions.
We conduct extensive experiments to validate the effectiveness of the proposed module and achieve sate-of-ther performance on five public vessel datasets covering three distinct imaging modalities.
2 Related Work
2.1 Foundation Models for Medical Image Segmentation
Foundation vision models such the SAM [T have demonstrated strong generalization and zero-shot capabilities in natural image segmentation. However, their performance degrades significantly in medical imaging tasks, particularly for fine-grained structures like vessels and retinal layers, due to domain shifts and lack of priors for thin, low-contrast anatomy [15] [6]. To bridge this gap, recent studies have explored adapter-basee based tuning 6 71, LoRA-based silent fine-tur tuning [8), automatic prompt generation [9], and hybrid approaches combining SAM with CLIP [O] as well as methods that improve boundary accuracy through high-quality priors and edge-aware refinement [7. While these methods improve global adaptability, they often overlook localized structural failures- vessel discontinuities, blurred edges, and fragmented predictions- -tht are critical in clinical applications [8]. FineSAM++ addresses this limitation by introducing a sparse residual expert framework guided by uncertainty-awant gating, enabling selective correction structurally uncertain regions while preserving global semantic consistency.
2.2 Mixture-of-Experrs Architectures in Vision Modeling
Mexture-of-Experrr (MoE) architectures have emerged powerful paradigm for scaling deep networks while maintaining efficiency [12] 13). In vision, recent works have explored various MoE formulations for different purposes. Switch Transformers [1] propose token-based routing
to conditionally activate expert blocks in large-scale transformers. Expert Choice Routing [14] and SwitchHead 220 further improve routing efficiency and stability by optimizing expert selection and
assignment. CuMo [21] introduces co-upeycled expert reuse to scale multimodal models, achieving strong performance with limited expert redundancy. Neural Experts (22) and related vision-specific MoE designs have focused primarily on large-scale classification and vision- languuge pretraining tasks. However, these works focus on token or patch-level routing for elassification or rision-languaae tasks, and have not explored dense prediction or fine-grat tne-grained structural correction. FineSAM++ to systematically integrate sparse expert routing into a dense segmentation pipeline. By introducing soft Gating Module and Residual Expert, FineSAM+ applies MoE principles to address local structural inconsistencies medical vessel segmentation, which remains largely underexplored in prior vision MoE literature.
3 Method
Inspired by the success of sparse MoE in scaling LLMs and vision models 12
13
we propose
FineSAM++, a sparse expert framework designed for fine-gu -reiigraded targets. Fine- line-grained target segmentation naturally fits the MoE paradigm due to the extreme sparsity, topology irregularity. and strong locality of error-prone regions. Our framework incorporates two specialized lightweight experts: a global LoRA Expert for domain adaptation and Residual Expert for structure-aware local residual correction. A differentiable Gating Module inspired by Expert Choice Routing [2] coordinates dynamic activation of experts.
3

---

Coarse Mask & T'mage
Router Network
Light CNN
LoRA-SAM
3G0
Degrade Fsaw. ,/))
Router Mapping
Gating Module
1.
fa inactive
f0
Expert
Residual Expert
Σ w r()
y = SSAM m O
Output
Figure 3: Overview of the FineSAM++ architecture.
3.1 Overview
Given an input image I E H x3, FineSAM++ first generates coarse segmentation prediction
ÚSAM E [0, || using LoRA-SAM model with degraded inputs (see Sec. 3.3 for details). Here, LoRA-SAM refers to a frozen SAM backbone augmented with LoRA adapters, following the parameter-efffiiiett fine-tuning strategy in [2]. Additionally, for the prompt encoder in LoRA-SAM. FineSAM+ does not require any manual prompts; instead, it learns fixed default embedding during training. While the coarse prediction is generally effective, it often exhibits topological inconsistencies in challenging regions such as vascular bifurcations, blurred boundaries, and disconnected thin structures.
To address this, FineSAM+ introduces a modular sparse refinement pathway consisting of (1) Gating Module to estimate spatial uncertainty and expert routing weights, and (2) J lightweight Residual Experts to perform localized residual correction (see 3)g Each Residual Expert receives perturbed variant of the coarse mask to promote specialization. The final output is obtained by fusing the original SAM prediction with the aggregated residual corrections:
Σ wj r(ti) )
i = o ŪSAM + m m
where m is the uncertainty mask, wy are routing weights, and (0ر are the expert outputs. @ denotes element-wise multiplication.
3.2 Gating Module
Classical MoE architectures rely token-level routing based dense embedding vectors 12
19
However, dense vision transformers like SAM produce structured 2D feature maps, where localized spatial uncertainty plays critical role. To address this gap, FineSAM++ introduces spatially- aware soft routing mechanism via dedicated Gating module. Our Gating module serves two purposes: (1) generate a soft mask
[0. 1] x indicating spatial uncertainty at each pixel, and (2) output fractional routing weights {w,} for J parallel Residual Experts. The module receives the concatenation of the image x and coarse mask ÚSAM as input, capturing both appearance and prediction context:
[wy} (Concat(z,, {SAM)).
(2)

---

This design is fundamentally different from standard MoE routers, which treat each input as inde- pendent. Instead. our Gating module explicitly leverages spatial correlations, identifying localized regions that require residual correction. By assigning soft weights to multiple experts, the routing mechanism enables fine-graa fine-grained specialization without hard top-k decisions, which are known to suffer from instability and expert imbalance [14 22
We supervise the Gating module using pseudo-labels derived from backbone prediction errors. binary pseudo-label mask 9t. is first generated by thresholding the absolute error between the SAM
coarse prediction ÚSAM and the ground truth label y:
( = (lÜSAM v()) 8);
where I(.) is the indicator function that outputs if the condition holds and 0 otherwise, and is a pre-defined error threshold. The Gating module is trained to predict router map m [0, 1] where higher values indicate greater structural uncertainty. We optimize the Gating output using the standard binary cross-entropy (BCE) loss:
Lganc
N Σ log + (1 - 9: (x) - m(1))),
(4)
where N the total number of pixels. This loss encourages the Gating module to activate only in regions where the backbone prediction is structurally unreliable, while suppressing unnecessary expert invocation elsewhere.
3.3 Residual Experts
Conventional segmentation refinement frameworks either retrain the full model or introduce single auxiliary head for residual prediction 2 In contrast, FineSAMM+ proposes a diverse multi-expert residual correction scheme. Inspired by MoE principles of expert specialization, we deploy J
parallel Residual Experts f())) each receiving slightly degraded version of the coarse mask:
(G) USAM
Degrade(ijAM: 77)),
(5)
where Degrade(-) applies random masking, noise injection, or occlusion perturbations to promote input diversity. This novel perturbation-basee expert diversification allows each expert to specialize in correcting specific structural failures, such as disconnections, thin vessel loss, or noisy edges.
((i) Each expert predicts residual corrections conditioned on both the perturbed coarse mask 9SAM and input image x::
v = _ USAM):
where v() E R HxW denotes the residual correction map predicted by the j-th expert. This formu- lation allows each expert to focus on different perturbation patterns and promotes specialization. Unlike prior works that produce complete segmentation masks, experts focus exclusively local residual correction. We supervise this process using a gated mean squared error (MSE) loss between the fused expert output and the ground truth:
Les = N
1
Σ (m Σ wy . ,())(()) + (DSAM(1) u(i))
where N is the total number of pixels, m(i) E [0, 1] is the soft spatial uncertainty mask from the Gating Module at pixel wy are the normalized routing weights for each expert, and y(i) is the ground truth label.
3.4 Progressive Optimization with Dynamic Weighting
Training strategy with two-phase adaptive learning. The Residual Expert strongly depends on stable backbone predictions to avoid overfitting to noisy residual targets. Therefore, we design two-phase adaptive training strategy. In Phase 1, we freeze both the Gating Module and Residual Expert and train only the SAM backbone until LSAM falls below a threshold for & consecutive epochs. This warm-up stage stabilizes the coarse prediction output.
5

---

Once convergence is detected, Phase 2 activates the residual correction pathway and applies dynamic loss weighting (ox
A (t) min (1, 1 60) (1. TT)
AtAM(() = 1 Aros(t)) (t),
(8)
where to is the warm-up completion epoch and T controls the progressive ramp-up of the Residual Expert contribution. This two-phase curriculum minimizes early instability and ensures smoother joint optimization of both the backbone and expert modules.
Overall Loss. The final training objective of FineSAM++ combines the semantic segmentation loss of the SAM backbone and the structural refinement losses of the Residual Expert:
L = ASAM Lsam + XresCres + Agate Ligate,
where LSAM represents the combined Dice and binary cross-entropy loss on the backbonn output, fres is the masked residual regression loss, and Lyme is the gating supervision loss. The weighting coefficients À balance the contributions of each component and are dynamically adjusted as described above. This formulation enables FineSAM++ jointly optimize global semantic consistency and local structural refinement in a stable and interpretable manner.
Table 1: Quantitative comparison on DRIVE, DCAI, CHUAC and ROSE datasets. The best results are bolded while the second best are underlined. Other Dataset (FIVES) quantitative comparison are provided in the supplementary material.
Data
Method
Dige
ACC
Metn:
AUC
SB
SP
Data
Method
Dise
ACC
Metric
AUR
SE
SP
U-Net
D37787
039166
03861
0.7907
0.9792
U-Ner
0,7392
0.941
0.9801
0.7647
03831
U-Not
0.7808
0.9621
0.974
0.7931
0.9795
ti-Net
0.7511
0.9753
0.9834
0.7851
0.9861
U-Net++
0.9635
0.625
U-Net++
0.7766
0.9757
0.9860
0.7932
0.9557
R2U-NNt
08171
09556
0.784
0.7792
0.9813
CS-Nei
0,7790
0.9763
0.9889
0.7855
0.9667
TransUNet
0.7872
0.577
0.9722
0.7819
0.9788
VSSC Net
0,9700
0.9831
0.7728
0.6099
CAVIT
0,9700
0.9661
0.7924
0.9872
FRR-UNet
0.7736
0.9744
0.9897
0.8341
09631
MCDAU-Net
08129
0.9589
0.8215
0.9739
McdUNAS
0.7820
0.9800
0.8089
09508
Retina TransNet
0.7964
0.8836
0,7850
0.9821
G2VIT
0.7659
0,9761
0.9904
0.8377
0.9114
MRC-Net
0.969
0.9855
0.8200
0.9857
HRNet
0.7919
0.777
0.9899
0,007
0.9676
Gupta
0.9978
0.9677
0.5843
0,786
0.9824
Gupea at
0.9681
0.9911
0.883
0.9911
DRIVE
RETFound
0.8020
0.9699
0.8830
0,7795
0,9821
DCAL
RETFound
0.7945
0.9685
0.9923
0.8857
0.9672
anUnner
03220
0.969%
0 0,894
(18019
0.9862
neliner
0.8045
0.9584
0
08264
019879
SAM Appies
0.44%%
013311
01224
0.7577
0.9377
SAM Aapter
0.75K3
0.9127
0:0208
0.7822
(19836
H-SAM
0.6622
0.9485
0.7624
0,5808
0.9840
H-SAM
0.6374
0,9661
0,7810
0.5732
0.9887
AusxSAm
0.9414
0L9872
01836
AutaSAm
0.7175
0.9693
0.8489
0.7120
0.970
SAMed
0.610
0.150
0,9000
0,5070
SAMed
0.9540
0,9550
0.5750
0.9760
HQ-SAM
0.7978
0.9677
0.8824
0,8033
0,9624
HQ-SAM
0.7830
0.9770
0,8890
0. 7890
0.988
Ours
03831
0.770)
(19570
0.3366
0834
Ours
0.8127
0.9775
0.9931
036479
0.9872
U.NG
DC73
09744
03332
0.500
1.991
U-Ner
0.7116
0.8935
0:9218
0.7867
03:30
Aul U-Net
0.9911
0.9633
0.515
0.6420
0.9922
CS-Nei
0.7608
0,9152
0,9392
0,8631
0,1122
U-Net=+
0.9802
06669
0.6099
CB-Net
0.7511
0.9121
0.9292
CS-Net
0
0.996
0.747
0,6755
0.9918
COSFIRE
0.2517
0.9227
0.9286
VSSC Not
0.721
0.757
0,7892
0.9797
COOF
0,6606
0.8550
0.8689
FR-UNet
0.7543
0.9740
0.9786
0.7836
0.9867
ResUI-Nat
0.7461
0.9098
0.9252
McdUNAS
0.9456
0.9807
0,7829
0.9912
DUNet
0.7505
0.9118
0.9334
GIVIT
0,7612
0.9809
0.8588
0.7908
0.9950
ihnce-stage
0.7663
0.9179
0.9179
HRNet
0.7526
0.9811
0.906
0.7456
0.9906
OCTA-Net
0.7087
0.9182
0.9453
Gupta
0.7168
0.9799
0.7399
0,6728
Gupea at
0,7801
0.9164
0.8563
0.9199
CHUAC
RETRound
0.7636
0.9604
0.9044
0,7325
0.990
ROSE
RETRRondd
0.7136
0.9197
0.9337
0.8563
0.993
nnUneer
0.7814
0.9776
0.3842
0.7788
0.9896
aatiner
0.8270
0.3470
0.9310
03660
0.9940
SAM Aupler
0.7636
0.9784
0.9359
0.7583
0.9902
SAM Aupler
0.6316
0.8576
0.8451
0.6503
0.9601
H.SAM
0.6951
0.9707
0,8310
016758
0.9862
H-SAM
0.6968
018973
0.7965
0.6335
0.9595
AusoSAm
0.6833
0.9654
0.8614
0.7457
AutoSAM
0.6954
0.8349
0.8054
0.6557
0.984
SAMed
0,9530
0,7040
SAMed
0.8810
0.9570
HQ-SAM
0.7050
8940
0,8120
0.6760
0.9480
HQ-SAM
0.7520
0.9609
textMm.
9904
77400
0.9887
Ours
0.7768
0.9077
0.9951
0.7567
0.9932
Ours
0.5120
0.9883
0.9827
0.9485
0.9623
4 Experiments
4.1
Implementation Details
Dataset. We evaluate FineSAM++ across five publicly available vascular segmentation datasets spanning three imaging modalities. The DRIVE dataset (23] contains two-dimensional retinal fundus images with ground truth vessel masks. ROSE (24 provides retinal vessel segmentation from 2D optical coherence tomography angiography (OCTA) scans. FIVES 25 includes 800 high-resolution multi-disease color fundus photographs annotated for vessel structures. DCA1 (66 and CHUAC (27) are coronary angiography datasets containing fluoroscopic X-ray vessel images. We select these datascts to cover the full spectrum of challenges targeted by FAM+SAM++, including variations in

---

anatomical regions (retina vs. coronary arteries), imaging modalities (fundus photography, OCTA, X-ray angiography), and segmentation difficulties (low contrast, thin structures, fragmented vessels). Detailed dataset statistics and preprocessing steps are provided in the supplementary material.
Beyond vascular segmentation. To further assess cross-domain generalization, we additionally evaluate multi-class abdominal organ segmentation on the Synapse Multi-Organ CT dataset (eight organs), demonstrating that FineSAM++ maintains strong performance outside the vascular domain. Detailed dataset statistics and results are provided in the supplementary material.
Image
Label
Our
H-SAM
SAM Adapter
SAMed
CHUAC
DB134
Drive
Fives
Rose
Figure 4: Qualitative comparison of FineSAM++ against H-SAM, SAM Adapter, and SAMed across five datasets. FiM provides more continuous and complete vessel segmentation with fewer false positives and fragmentation artifacts.
Training settings. All experiments are implemented using Py and trained on two NVIDIA RTX 4090 GPUs Data augmentation includes random elastic deformation, rotation, scaling, and intensity jittering. For the backbone, we follow ( and integrate LoRA adapters into the frozen SAM encoder with rank of 4. We adopt the ViT-B configuration of SAM the base encoder. For fair comparison across datasets, all images are resized to 512 x 512 resolution. The maximum training epoch is set to 300. We use the A optimizer with learning rate is set to x 10-5 and decayed using a cosine annealing schedule. All hyperparameters fixed across datasets without additional tuning to ensure fair comparison and reproducibility.
0.9, 83 0.999, and weight decay of 0. The initial
Evaluation metrics. To comprehensively assess model performance, we evaluate FineS AM++ and all baselines across standard segmentation accuracy metrics and specialized structural consistency metrics tailored for vascular image analysis. The primary evaluation metrics include Dice, accuracy (ACC), sensitivity (SE), specificity (SP), connectivity (C), overlapping area (A), vessel length consistency (L), and centerline Dice (cIDice). For completeness and reproducibility, detailed definitions for all metrics are provided in the supplementary material.
Compared methods. this study, we comprehensively benchmark FineSAM++ against wide range of state-of-the-ant (SOTA) methods previously reported on the evaluated datasets. The baselines

---

include both classical CNN-based and modern Transformer-baat segmentation models, topology- aware segmentation method, as well as several recent SAM-variant foundation model adaptations. For clarity and reproducibility, the full list of compared methods and corresponding references are provided in the supplementary material.
Table 2: Quantitative comparison on DRIVE, FIVES, DCAL, CHUAC and ROSE datasets. Metrics include connectivity (C), area accuracy (A), length similarity (L), and centerline Dice (CIDice). The best results are bolded and the second best are underlined.
Data
Method
Metrics
Data
Method
Metrics
ClDice
C
CIDice
U-Net
0,998
0.712
0.608
0.761
U-Net
0.994
0.897
0.912
0.889
Att U-Net
0.994
0.768
0.732
0.775
Swin-Unet
0.997
0.863
0.871
0.785
U-Net++
0.994
0.783
0.774
0.801
TransUnet
0.997
0.919
0.923
0.911
SAM Adapter
0.993
0.412
0.321
0.488
SAM Adapter
0.993
0.713
0.654
0.878
DRIVE
H-SAM
0.997
0.631
0.654
0.612
FIVES
H-SAM
0.994
0.698
0.711
0.645
AuteSAM
0.998
0.597
0.621
0.598
AutoSAM
0.996
0.652
0.675
0.887
SAMed
0.996
0.583
0.561
0.556
SAMed
0.994
0.691
0.712
0.657
0.998
0.848
0.865
0,832
Ours
0.997
0.921
0.925
0.914
U-Net
0.995
0.78
0.812
0,7900
U-Net
0.996
0.723
0.739
0.7100
An U-Net
0.996
0.812
0.798
0,8050
CS-Net
0.997
0.776
0.789
0,7500
U-Net++
0
0.831
0.813
0.8150
OCTA-Net
0.999
0.781
0.765
0.7550
DCAI
SAM Adapter
H-SAM
0.992
0.998
0.785
0.732
0.734
0.812
0.72
0.8
ROSE
SAM Adapter
H-SAM
0.992
0.994
0.721
0.683
0.719
0.657
0.7050
0.6600
AutoSAM
0.995
0.732
0.757
0.735
AutoSAM
0.995
0.736
0.743
0.7150
SAMed
0.997
0.643
0.651
0.65
SAMed
0.997
0.675
0.674
0.6900
Ours
0.997
0.903
0.877
0.865
Ours
0.997
0,819
0.839
0.8050
U-Net
0.994
0.631
0.629
0.6200
H-SAM
0.994
0.712
0.719
0.7000
CHUAC
Att U-Net
U-Net++
0.996
0.995
0.702
0.723
0.698
0.722
0.7150
0,6850
CHUAC
AutoSAM
SAMed
0.997
0.993
0.757
0.698
0.723
0.739
0.7350
0.6900
SAM Adapter
0.998
0.759
0.768
0.75
Ours
0.998
0.787
0.795
0.7700
4.2
Main Results
Quantitative Comparisons. Tab. summarizes the performance comparison across four datasets. FineSAM++ consistently achieves superior results over both CNN- and Transformer-base baselines as well as recent SAM-derived methods. On DRIVE and DCAI, FineSAM++ sets new state-of-the-att Dice scores of 0.8231 and 0.8127, respectively, substantially outperforming AutoSAM (0.6603 and 0.7175) and H-SAM (0.6622 and 0.6374). On CHUAC and ROSE, our method also delivers the highest Dice scores (0.7768 and 0.8220), demonstrating robust generalization across diverse vascular modalities. These results validate the effectiveness of our multi-expert sparse refinement strategy in addressing localized structural failures of foundation segmentation models while maintaining high global consistency.
Qualitative Results. Fig. 4 shows representative qualitative comparisons of FineSAM+ against leading SAM-variant methods (H-SAM, SAM Adapter. SAMed) across five datasets. Our method consistently produces sharper and more continuous vessel structures with fewer false positives and disconnected branches. In coronary angiography datasets (CHUAC, DB134), FineSAM++ better captures thin vessel bifurcations and suppresses background noise. On retinal fundus images (DRIVE, FIVES), our model recovers small peripheral vessels missed by baselines. For OCTA images (ROSE), FineSAM++ yields smoother centerlines with significantly reduced fragmentation compared to prior approaches. These visual improvements highlight the advantage of our multi-expert sparse refinement design for addressing localized structural errors while preserving global topology.
Topological analysis. Tab. 7 reports connectivity (C), area (A), length (L), and CIDice metrics across four datasets. FineSAM++ consistently achieves the highest CIDice scores, indicating superior preservation of vessel topology and centerline continuity. On DRIVE and DCAL, our method outperforms the strongest baseline by margins of 0.832 vs. 0.801 and 0.865 vs. 0.815 respectively. Similar trends are observed on CHUAC and ROSE. The strong gains in connectivity (C) and length (L) further highlight the advantage of our sparse expert refinement design in correcting disconnections and fragmented vessels present in the coarse backbone predictions. These results demonstrate that FineSAM++ not only improves segmentation accuracy but also enhances structural fidelity, which is critical in clinical vascular analysis.
8

---

Image
Label
Lora-SAM
Gating Map
Expert
Our
Figure 5: Visualization of the FineSAM+ refinement process.
4.3 Ablation Study
Parameter Efficiency:To address concerns about the parameter efficiency of our proposed method FineSAM+ among SAM-based segmentation approaches, we conducted a comprehensive compari- son under standard input resolution of (1,3, 1024, 1024). Specifically, we measured the number of parameters, FLOPs (GAMCs), and inference latency, as summarized in Table 3 As shown, Fine- SAM++ introduces only about 0.7M additional learnable parameters over the SAM backbone (94.4M 93.7M. 7.) Compared with other 3AM-based methods-suut as SAM Adapter (104.3M), H-SAM (111.3M), and AutoSAM (135.9MM)-iimtn FineSAM++ demonstrates substantially higher parameter efficiency. Moreover, while its total parameter count is higher than lightweight architectures like Unet, FineSAM++ achieves the highest Dice score (0.8231) among all evaluated methods. These results indicate that FineSAM++ achieves excellent balance between parameter efficiency and segmentation performance.
Table Comparison of model size, computational cost, latency, and segmentation accuracy (Dice
score) across segmentation methods using an input of size (1, 3, 1024, 1024).
Model
Params (M)
GAMC's (G)
Latency (ms)
Dice
Unets
34.53
4.08
1.10
0.7787
nnUnet
126.2
1864.9
37.4
0.8220
SAM Adapter
104.3
400.1
127.8
0.4498
H-SAM
111.3
370.6
124.8
0.6622
AutoSAM
135.29
774.16
166.22
0.6603
SAMed
92.2
370.5
117.1
0.6170
SAM
93.7
372.0
116.33
Ours (FineSAM++)
94.4
376.8
117.6
0.8231
Ablation Study on the Gating Threshold 8. To assess the sensitivity of the Gating module to the pre-defined error threshold 6, we conduct an ablation on the DRIVE dataset by varying 5 E (0.3, 0.4, 0.5, 0.6, 0.73. with results summarized in Table 4 While certain metrics (c.g., SE at

---

5 = 0.3) are slightly higher, ö = 0.5 delivers the best overall performance across Dice, ACC, AUC, SE, and SP. Intuitively, too-small threshold treats most pixels as uncertain, triggering unnecessary refinement and reducing gating sparsity, whereas too-large threshold routes only a few pixels and underutilizes the refinement experts. Balancing these effects, we adopt fixed threshold of 0.5 across all five public datasets for both accuracy and efficiency.
Table 4: The ablation results of threshold 6 in the Gating module.
5
Dice
ACC
AUC
SE
SP
0.3
0.8124
0.9712
0,9812
0.8432
0.9601
0.4
0.8187
0.9755
0.9846
0.8410
0.9732
0.5
0.8231
0.9790
0.9870
0.8366
0.9834
0.6
0.8180
0.9767
0.9854
0.8204
0.9807
0.7
0.8129
0.9735
0.9822
0.8083
0.9784
Effect of number of Residual Experts. We evaluate the effect of varying the number of Residual Experts (J = 1,2, 4, 6) on the DRIVE dataset, as shown in Tab Increasing j consistently improves segmentation performance. The Dice score rises from 0.7501 (1 expert) 0.8231 (6 experts), with corresponding gains across all other metrics. Notably, performance gains begin to saturate beyond j 4. suggesting that using moderate number of experts balances accuracy and computational efficiency. We adopt j = 4 for all remaining experiments as trade-off between performance and resource consumption.
Table 5: Ablation study of FineSAM+ modules on the DRIVE dataset.
Lora- LAMa-SAM
Gating
Residual Experts
Dice
ACC
AUC
SE
SP
0.7322
0.9524
0.9711
0.7865
0.9678
0.7871
0.9634
0.9821
0.8147
0.9772
V
0.8231
0.9790
0.9870
0.8366
0.9834
Qualitative Analysis of Local Structure Refinement. Fig. presents the FineSAM++ refinement pipeline across multiple vascular segmentation datasets, including fundus, angiography, and OCT-like images. Starting from the coarse LoRA-SAM prediction, which often suffers from topological errors, the Gating Module identifies uncertain regions and selectively routes them to Residual Experts for localized correction. The final output shows improved connectivity and boundary completeness. For visualization clarity, only the first expert's outputs are shown, though FineSAM++ operates with mixture of experts.
Table 6: Ablation study of the number of Residual Experts (J) on the DRIVE dataset.
Number
Dice
ACC
AUC
SE
SP
0.7501
0.956
0.9751
0.7943
0.9723
0.7769
0.9621
0.9803
0.8081
0.9763
0,8025
0.9655
0,9811
0.8139
0.9783
0.8231
0.9790
0.9870
0.8366
0.9834
Ablation study of modules. Tab, 5 presents the effect of incrementally adding FineSAM+ com- ponents. Using only the LoRA-SAM backbone yields limited performance (Dice 0.7322). Adding Residual Experts without the Gating Module, where all J experts are uniformly averaged without spatial weighting, improves performance 0.7871 Dice by introducing localized correction. How- ever, enabling the full pipeline with the Gating Module further increases performance to 0.8231 Dice and leads consistent improvements across all metrics. The Gating Module provides soft spatial routing mechanism that dynamically assigns different weights to each expert's output based on local uncertainty, promoting expert specialization and sparse activation. These results validate that targeted soft routing is critical for maximizing expert effectiveness and minimizing unnecessary corrections in confident regions.
5 Conclusion
We presented FineSAM++, structure-aware sparse expert framework for enhancing foundation seg- mentation models in fine-grained medical image analysis. By introducing soft Gating Module with
10

---

uncertainty-awarr aware spatial routing and deploying multiple Residual Experts with input perturbation diversity, our method achieves localized structural refinement while maintaining global consistency. Extensive experiments on five vascular segmentation benchmarks demonstrate that FineSAM++ consistently outperforms both classical and SAM-adapted baselines across accuracy and topological continuity. Our results validate the effectiveness of sparse expert activation for addressing localized segmentation failures. Future work will explore dynamic expert allocation, adaptive perturbation strategies, and generalization to other medical and natural image dense prediction tasks.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 82302300). National Natural Science Foundation of China (62376231), Sichuan Science and Technology Program (2024NSFC0658)
References
[1] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., "Segment anything." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026 2023.
[2] S. Gong, Y. Zhong, W. Ma, J. Li, Z. Wang, J. Zhang, P.-A. Heng, and Q. Dou, "3dsam-adapter. Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation," arXiv preprint arXiv:2306. 3365,, 2023.
[3] Ma, Y. He, Li, L. Han, C. You, and B. Wang. "Segment anything in medical images." Nature Communications, vol. 15, no. 1, p. 654, 2024.
[4] Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen, el al., "Segment anything model for medical images?," Medical Image Analysis, vol. 92, p. 103061, 2024.
[5] Z. Qiu, Y. Hu, H. Li, and J. Liu, "Learnable ophthalmology sam," arXiv preprint arXiv:2304.13422 13425, 2023.
[6] J. Zhu, A. Hamdi, Y. Qi, Y. Jin, and Wa, "Medical sam 2: Segment medical images as video via segment anything model 2," arXiv preprint arXiv:2408.00877 2024.
[7] J. Wu, 7. Wang, M. Hong, W. Ji, H. Fu, Y. Xu, M. Xu, and Y. Jin, "Medical sam adapter: Adapting segment anything model for medical image segmentation," Medical image analysis, vol. 102, p. 103547, 2025.
[8] Z. Cheng, Q Wei, H. Zhu, Y. Wang. L. Qu, W. Shao, and Y. Zhou, "Unleashing the potential of sam for medical adaptation via hicrarchical decoding." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3511-3522, 2024.
[9] C. Li, R. I. Sultan, P. Khanduri, Y. Qiang. C. Indrin, and D. Zhu, "Autoprosam: Automated prompting sam for 3d multi-organ segmentation." " in 2025 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 3570-3580, IEEE, 2025.
[10] X. Yan, S. Sun, K. Han, T.-T. Le, H. Ma, C. You, and X. Xie, "After-sam: Adapting sam with axial fusion transformer for medical imaging segmentation," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7975-7984, 2024.
[11] S. Aleem, F. Wang, M. Maniparambil, E. Arazo, J. Dietlmeier, K. Curran, N. E. Connor, and S. Little, "Test-time adaptation with salip: A cascade of sam and clip for zero-shot medical image segmentation," in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5184-5193, 2024.
[12] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, "Outra- geously large neural networks: The sparsely-gate mixture-of-experts layer," arXiv preprint arXiv:1701.06538. 2017.
11

---

[13] C. Riquelle, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, "Scaling vision with sparse mixture of experts," Advancces in Neural Information Processing Systems, vol. 34, pp. 5583-8599 2021.
[14] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, "Gshard: Scaling giant models with conditional computation and automatic sharding," arXiv preprint arXiv:2006. 16668, 2020.
[15] J. Huang, K. Jiang, J. Zhang. H. Qiu, L. Lu, S. Lu, and E. Xing, "Learning prompt segment anything models," arXiv preprint arXiv:2401. 104461,,,1, 2024.
[16] Z. Qin, H. Yi, Q. Lao, and K. Li, "Medical image understanding with pretrained vision language models: A comprehensive study," arXiv preprint arXiv:2209. 15519, 2022.
[17] L.. Ke, M. Ye, M. Danelljan, Y.W. Tai, C.-K. Tang. F. Yu, el al., "Segment anything in high quality," Advances in Neural Information Processing Systems, vol. 36, pp. 29914-29934, 2023.
[18] Y. Zhang, Z. Shen, and R. Jiao, "Segment anything model for medical image segmentation: Current applications and future directions," Computers in Biology and Medicine, p. 108238, 2024.
[19] W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," Journal of Machine Learning Research, vol. 23, no. 120, pp. 1-39, 2022.
[20] R. Csordás, P. Piçkos, K. Irie, and J. Schmidhuber, "Switchhead: Accelerating transformers with mixture-of-experts attention," Advances in Neural Information Processing Systems, vol. 37, pp. 74411-74438, 2024.
[21] J. Li, X. Wang, S. Zhu, C.-W. Kuo, L. Xu, F. Chen, J. Jain, H. Shi, and L. Wen, "Cumo: Scaling multimodal IIm with co-upcycled mixture-of-expeee -fpeeerrtss""""""""" Advances in Neural Information Processing Systems, vol. 37, pp.
2024.
[22] Y. Ben-Shabat, C. Hewa Koneputugodage, S. Ramasinghe, and S. Gould, "Neural experts: Mix- ture of experts for implicit neural representations," Advances in Neural Information Processing Systems, vol. 37, pp. 101641-101670, 2024.
[23] Staal, M. D. Abråmoff, M. Niemeijer. M. Viergever, and B. Van Ginneken, "Ridge-based vessel segmentation in color images of the retina," IEEE transactions on medical imaging. vol. 23, no. 4, pp. 501-509, 2004.
[24] Y. Ma, H. Hao, Xie, H. Fu, J. Zhang, J. Yang, Z. Wang, J. Liu, Zheng, and Y. Zhao, "Rose: a retinal oct-angiography vessel segmentation dataset and new model," IEEE transactions on medical imaging, vol. 40, no. 3, pp. 928-939, 2020.
[25] K. Jin, X. Huang, J. Zhou, Y. Li, Y. Yan, Y. Sun, Q. Zhang, Y. Wang, and Ye, "Fives: A fundus image dataset for artificial intelligence based vessel segmentation." Scientific Data, vol. 9, no. 1, p. 44457, 2022.
[26] D. Hao, S. Ding, L. Qiu, Y. Lv, B. Fei, Y. Zhu, and B. Qin, "Sequential vessel segmentation via deep channel attention network," Neural Networks, vol. 128, pp. 172-187, 2020.
[27] F. Cervantes-Sanchez, I. Cruz-Aceves, A. Hernandez-Aguirre, M. A. Hernandez Gonzalez, and S. E. Solorio-Meza, "Automatic segmentation of coronary arteries in x-ray angiograms using multiscale analysis and artificial neural networks," " Applied Sciences, vol. 9, no. 24, p. 5507, 2019.
[28] H. Zhang, Z. Gao. D. Zhang, W. K. Hau, and H. Zhang, "Progressive perception learning for main coronary segmentation in x-ray angiography," IEEE Transactions on Medical Imaging, vol. 42, no. 3, pp. 864 879, 2022.
[29] B. Landman. Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein, "Miccai multi-atlas labeling beyond the cranial worlt-wo and challenge," in Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Workshop Challenge, vol. 5, p. 12, 2015.
12

---

[30] J. Chen, Y. Lu, Q. Yu, X. Luo. E. Adeli, Y. Wang. L. Lu, A. L. Yuille, and Y. Zhou, "Tran- sunet: Transformers make strong encoders for medical image segmentation," " arXiv preprint arXiv:2 04306 2021.
[31] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation." in Medical image compuling and computer-assisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241, Springer, 2015.
[32] O. Oktay, J. Schlemper, L. L. Folgoe, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, et al., "Attention u-net: Learning where to look for the pancreas," arXiv preprini arXiv: arXiv:/804.0000 1804,03999, 2018.
[33] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, "Unet++: A nested u-net architecture for medical image segmentation," in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings pp. 3-11, Springer, 2018
[34] L. Mou, Y. Zhao, L. Chen, J. Cheng, Z. Gu, H. Hao, H. Qi, Y. Zheng, A. Frangi, and J. Liu, "Cs- net: Channel and spatial attention network for curvilinear structure segmentation." in Medical Image Computing and Computer Assisted IntervntionnMICCAA 2019 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part 22, pp. 721-730, Springer, 2019.
[35] P. M. Samuel and T. Veeramalai, "Vssc net: vessel specific skip chain convolutional network for blood vessel segmentation," Computer methods and programs in biomedicine, vol. 198, 105769, 2021.
[36] W. Liu, H. Yang, T. Tian, Z. Cao, X. Pan, W. Xu, Y. Jin, and F. Gao, "Full-resolution network and dual-threshold iteration for retinal vessel and coronary angiograph segmentation," IEEE journal of biomedical and health informatics, vol. 26, no. 9, pp. 4623-4633 2022.
[37] R. Li, Zheng, C. Duan, J. Su, and C. Zhang, "Multistage attention resu-net for semantic segmentation of fine-resolution remote sensing images," IEFE Geoscience and Remote Sensing Letters vol. 19, pp. 1-5, 2021.
[38] Z. Kuş and B. Kiraz, "Evolutionary architecture optimization for retinal vessel segmentation," IEEE Journal of Biomedical and Health Informatics, 2023.
[39] Q. Jin, Z. Meng. D. Pham, Q. Chen, L. Wei, and R. Su, "Dunet: deformable network for retinal vessel segmentation," Knowledge-Based Systems, vol. 178, pp. 149-162, 2019.
[40] K. Sun, B. Xiao, D. Liu, and Wang, "Deep high-resolution representation learning for human pose estimation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5693-5703, 2019.
[41] M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, and K. Asari, "Recurrent residual convolu- tional neural network based on u-net (r2u-net) for medical image segmentation," arXiv preprint 1802.068339 2018
[42] L. Yao, C. Mao, and Y. Luo, "Graph convolutional networks for text classification," in Proceed- ings of the AAAI conference on artificial intelligence, vol. 33, pp. 7370-7377, 2019.
[43] L.-C. Chen, Y. Zhu, G. Papandreou, Schroff, and H. Adam, "Encoder-decoddr with atrous separable convolution for semantic image segmentation," in Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018
[44] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "Cham: Convolutional block attention module," in Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
[45] H. Zhao, J. Shi, X. Qi, X. Wang, and Jia, "Pyramid scene parsing network," Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2881-2890, 2017.
13

---

[46] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, "Enet: A deep neural network architecture for real-time semantic segmentation." " arXiv preprint arXiv:1606.021/ 021477 2016.
[47] W. Wu, Y. Zhang, D. Wang, and Y. Lei, "Sk-net: Deep learning on point cloud via end-to-end discovery of spatial keypoints," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 6422-6429, 2020.
[48] V. Badrinarayanan, A. Kendall, and R. Cipolla, "Segnet: A deep convolutional encoder-decoder architecture for image segmentation," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 12, pp. 2481-2495, 2017.
[49] G. Azzopardi and N. Petkov, "Trainable cosfire filters for keypoint detection and pattern recognition." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. pp. 490-503, 2012.
[50] 7. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu, "Ce-net: Context encoder network for 2d medical image segmentation," " IEEE transactions on medical imaging, vol. 38, no. 10. pp. 2281-2292, 2019.
[51] J. Zhang, Y. Qiao, M. S. Sarabi, M. M. Khansari, J. K. Gahm, A. H. Kashani, and Y. Shi, "3d shape modeling and analysis of retinal microvasculature in oct-angiography images," IEEE transactions on medical imaging, vol. 39, no. 5, pp. 1335-1346, 2019..
[52] W. Zhou, W. Bai, J. Ji, Y. Yi, N. Zhang, and W. Cui, "Dual-path multi-scale context dense aggregation network for retinal vessel segmentation," Computers in Biology and Medicine, vol. 164, p. 1022.
[53] T. M. Khan, S. Naqvi, A. Robles-Kelly, and Razzak, "Retinal vessel segmentation via a multi-resolution contextual network and adversarial learning," Neural Networks, vol. 165, pp. 310-320, 2023.
[54] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang, "Swin-unet: Unet-like pure transformer for medical image segmentation," in European conference computer vision, pp. 208-218, Springer, 2022.
[55] J. Lin, X. Huang, H. Zhou, Y. Wang. and Q. Zhang, "Stimulus-guided adaptive transformer network for retinal blood vessel segmentation in fundus images," Medical Image Analysis, 102929, 2023.
[56] H.-C. Shao, C.-Y. Chen, M.-H. Chang, C.-H. Yu, C.-W. Lin, and J.-W. Yang, "Retina-transnet: a cundient-guided few-shot retinal vessel segmentation net," IEEE Journal of Biomedical and Health Informatics, 2023.
[57] M. S. Hossain, M. M. Rahman, M. M. Syeed, U. H. Hannan, M. F. Uddin, and S. B. Mumu, "Cavit: Early stage dental caries detection from smartphone-image using vision transformer," in 2023 4th International Conference Artificial Intelligence, Robotics and Control (AIRC), pp. 9-14, IEEE, 2023.
[58) H. Xu and Y. Wu, "G2vit: Graph neural network-guided vision transformer enhanced network for retinal vessel and coronary angiograph segmentation," Neural Networks, vol. 176, p. 106356, 2024.
[59] T. Chen, L. Zhu, C. Deng, R. Cao, Y. Wang, S. Zhang, Z. Li, L. Sun, Y. Zang, and P. Mao, "Sam-adapter: Adapting segment anything in underperformed scenes," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3367-3375, 2023.
[60] X. Hu, X. Xu, and Y. Shi, "How to efficiently adapt large segmentation model (sam) to medical images," " arXiv preprint arXiv:2306. / 13731, 2023.
[61] S. Gupta, Y. Zhang, X. Hu, P. Prasanna, and C. Chen, "Topology-awee aware uncertainty for image segmentation," Advances in Neural Information Processing Systems, vol. 36, pp. 8186-8207, 2023.
14

---

A Dataset Details and Evaluation Metrics
A.1 Dataset Details
We conduct experiments five publicly available medical segmentation datasets. The 2D datasets
include DRIVE 23, ROSE (24), FIVES [25]. DCAI [26]. and CHUAC (27).
23 The DRIVE dataset consists of 40 retinal fundus images (584 x 565 pixels) for vessel DRIVE
segmentation. We follow the official split of 20 training and 20 testing images.
ROSE [24]. The ROSE dataset contains 2D retinal optical coherence tomography angiography
(OCTA) scans. We use the ROSE-1 (SVC) subset comprising 30 training and testing images
(304 x 304 pixels).
The FIVES dataset (Fundus Image Vessel Segmentation) provides 800 high-resolution color fundus images (2048 x 2048 pixels) with pixel-level vessel annotations. The dataset is split into 600 training and 200 testing images.
FIVES 25
DCAI 26. The DCAI dataset contains 134 coronary angiography images (300 x 300 pixels). We follow the dataset's 's standard split with 100 training and 34 testing images.
The CHUAC dataset consists 30 coronary angiography images (189 189 pixels) with vessel annotations. Following [28), we split the dataset into 20 training and 10 testing images.
CHUAC 27).
Synapse Dataset. The dataset [29 contains 30 subjects for training and 20 subjects for testing with abdominal CT scans. consists of 13 organs, including organs of Synapse, along with esophagus, inferior vena cava, portal and splenic veins, right and left adrenal gland. Consistent with the partitioning strategy outlined in (30
A.2 Compared Methods
We compare FineSAM+ against comprehensive set of state-of-the-art (SOTA) methods previously reported on the evaluated datasets. The competing approaches are categorized into three groups: CNN-based segmentation models, Transformer-basee segmentation models, and foundation model variants.
(1) CNN-based methods. Classical and recent CNN architectures include U-Net [3]] Attention U-
Net 32). U-Net++ [33]. CS-Net (34), VSSC Net [35). FR- FR-UNet [36), R-sU-Net [37], MedUNAS 38]
DUNet 33. HRNet 40) R2U-Net 4T], GCN
42). Deeplab V3+ 43 CBAM 44 PSPNet 45
ENet 46 SK-Net 47). SegNet [8]. COSFIRE
49 CE-Net [0]. OCTA-Net 24 COOF S1
MCDAU-Net 52 and MRC-Net
53
(2) Transformer-based methods. Recent hybrid or fully Transformer architectures include Swin-
Unet [54) TransUNet 300 SGAT-Net (55). Retina-TransNet (56), CAViT [57]. and G2VIT (58).
(3) SAM foundation model variants. To benchmark against foundation model-based baselines, we
include SAM Adapter (59), H-SAM 88, AutoSAM [60]. SAMed (3]
(4) Topology-aware segmentation. the topology-aware adaptation by Gupta al. 61
B Evaluation metrics
To comprchensively assess the model's performance, we introduce the following evaluation metrics: Dice cocfficient (Dice), accuracy (ACC), sensitivity (SE), specificity (SP).
Dice. DICE score is a popular metric which measures the area/volumetric overlap between the predicted and ground truth discrete masks. It overcomes the class imbalance problem in the pixel- wise accuracy metric by considering only the foreground classes for measuring the overlap. The higher the DICE, the better the segmentation.
Accuracy (ACC). ACC measures the overall correctness of the segmentation results, calculating the proportion of correctly classified pixels or voxels to the total number of pixels or voxels.
15

---

Sensitivity (SE). SE also known as true positive rate or recall, quantifies the model's ability to correctly identify positive instances, indicating the proportion of true positives correctly classified among all actual positives.
Specificity (SP). SP measures the model's ability correctly identify negative instances, representing the proportion of true negatives correctly classified among all actual negatives,
Connectivity (C). Connectivity evaluates the structural consistency between the predicted segmen- tation and the ground truth. It measures the extent to which the connectivity of predicted regions matches that of the ground truth, ensuring the preservation of continuous structures, particularly in medical images.
Overlapping Area (A). Overlapping Area measures the absolute area of intersection between the predicted segmentation and the ground truth. Unlike IOU, it focuses solely on the shared region size, often serving supplementary metric for segmentation overlap evaluation.
Consistency of Vessel Length (L). quantifies the similarity in vessel lengths between the pre- dicted segmentation and the ground truth. This metric is particularly critical in vascular structure segmentation, ensuring that the predicted vessels maintain accurate geometric proportions.
elDice. A topology-base based metric is particularly sensitive to a model's performance on thin structures. This metric evaluates the overlap between predicted and ground truth masks while incorporating the topological features of the segmentation output.
C Experiments Results
C.1 Result on Vessel Segmentation
In this section, we add quantitative comparison on FIVES datasets. As shown in Tables 7 it can be seen that our method has relatively higher evaluation indicators.
Table 7: Quantitative comparison FIVES datasets. The best results are bolded while the second best are underlined.
Metric AUC
Method
Dice
ACC
SP
IOU
U-Net
0.8887
0,9866
0.9300
0.9910
0,8077
R2U-Net
0.8492
0,9809
0.9238
0.9899
0.7465
Att Unet
0.8881
0.9868
0.9272
0.9907
0,8073
GCN
0.90012
0.9879
0.9399
0,9922
0.8260
Deeplab V3+
0,8856
0.9850
0.9485
0.9933
0,8075
SK
0,8835
0.9858
0.9334
0,9912
0.7994
CBAM
0,8850
0.9867
0.9226
0.9901
0,8029
PSPNet
0.8988
0,9878
0.9396
0.9920
0,8235
ENet
0.8909
0.9867
0.9409
0.9922
0,81 10
SegNet
0,8509
0.9813
0.9244
0.9899
0.7498
Swin-
0,9013
0,9882
0,9402
0,9922
0.8276
TransU-Net
0.9037
0,9883
0,9447
0.99 0,9928
0,8317
SGAT-Net
0,9051
0,9886
0,9467
0. 0.993
0,8347
SAM Aapter
0.6313
0.8578
0.8451
0.9081
0,4630
H-SAM
0.6696
0.9603
0.7851
0.9887
0.5077
AutoSAM
0.8817
0.9875
0.9843
0.9921
0.7979
SAMed
0.6750
0,9590
0.9720
0.9840
0.5140
Ours
0.9141
0.9963
0.9961
0,9939
0.8258
C.2 Generalization beyond vessel segmentation
To validate that our framework generalizes beyond vessel segmentation, we evaluate multi-class abdominal organ segmentation on the Synapse Multi-Organ CT dataset (eight organs). Following prior work 30 3 g we adopt the standard split of 18 training and 12 test volumes and apply the corresponding preprocessing and augmentation protocols. As summarized in T our method attains the highes/ mean Dice (87.9 (87.97%) and the lowest Hausdorff Distance (HD: 7.89) among all compared methods, surpassing strong baselines such as H-SAM and nnU-Net. Notably, our approach maintains high accuracy challenging small structures (e.g., pancreas), indicating that FineSAM++
16

---

preserves strong segmentation quality and robustness when extended to more delicate anatomical targets.
Table 8: Comparison with ofate-of-the-ar art models on the Synapse orgai CT dataset.
Method
Spleen
Right Kidney
Left Kidnay
Gallbladder
Liver
Stomach
Aorta
Pancrvas
Mean Dice
HD
TNetsUNet
87.23
63.13
81.87
77.02
94.08
55.88
85.08
75.62
77.48
31.69
SwinUNet
85.47
66.53
83.28
79.61
94.29
56.58
90.66
76.60
79.13
21.55
TransDeeplab
DAE-Former
88 96
72.30
86.08
80.83
94.98
65.12
91 .4
79.19
82.43
17.46
MERIT
92.01
84.85
87.79
74.40
95.26
85.38
87.71
71.81
84,90
13.22
net-Net
AutoSAM
80.54
80.02
79,60
41.37
89.24
61.14
82.56
44,22
62.08
27.56
SAM Adapter
83.68
79.00
79.02
57.49
92.67
69.48
77.93
43.07
72,80
33,08
SAMed
69.11
79.95
72.17
20, 60.64
H-SAM
93.34
89.93
91.88
73.49
95.72
87. $7.10
89.38
71.11
36.49
8.18
Ours
94.25
91.53
93.21
71.23
96.89
90.83
92.52
82,23
87.97
7,89
D Ablation study
D.1 Effect of Progressive Optimization with Dynamic Weighting
We evaluate the impact of progressive optimization with dynamic uncertainty- based loss weighting by comparing three training strategies: (i) Naive Joint Training, i.e., end-to-end optimization with uniform loss weights; (ii) Stage-wise (Independent), which freezes the coarse module and trains refinement module separately; and (iii) Ours (Progressive), which progressively optimizes the two modules with dynamic, uncertainty-awarr weighting. As summarized Table the progressive strategy consistently achieves the best overall performance across Dice, ACC, AUC, SE, and SP. We attribute these gains to tighter interaction between the coarse and refinement modules and the reweighting of supervision toward uncertain regions, which improves refinement quality without overfitting.
Table 9: Ablation study on training strategies.
Strategy
Dice
ACC
AUC
SE
SP
Naïve Joint Training
0.7984
0.9641
0.9745
0.8123
0.9632
Stage-wise (Independent)
0.8117
0.9722
0.9810
0.8289
0.9766
Ours (Progressive)
0.8231
0.9790
0.9870
0.8366
0.9834
D.2 Degrade Strategy for Robust Multi-Expert Refinement
Refinement modules may overfit thin structures, which can increase false positives degrade mask quality in regions with weak boundaries or ambiguous textures. To mitigate this, we introduce Degrade strategy in FineSAM++: rather than feeding all experts the same coarse mask. apply randomized degradations to the coarse mask (see Eq. (5)) to encourage input diversity and expert specialization. This promotes complementary expertise across residual experts and improves robustness to structural uncertainty. As summarized in Table [T0 enabling the Degrade strategy yields consistent gains across Dice, ACC, AUC, and SP, while maintaining competitive SE, indicating fewer false positives and stronger generalization.
Table 10: Ablation study of the Degrade strategy for multi-expert training.
Strategy No Degrade
Dice
ACC
AUC
SE
SP
0.8154
0.9735
0.9813
0.8281
0.9720
Degrade
0.8231
0.9790
0.9870
0.8366
0.9834
E Statistical significance.
We assess the statistical significance of our improvements using paired 1-tests between our method and each baseline across all test images and datasets. Table [ reports the resulting p-values for five
17

---

metrics (Dice, ACC, AUC, SE, SP); values in bold indicate < 0.05. As shown, the majority of comparisons reach statistical significance, supporting that our approach yields superior segmentation accuracy with improved consistency (lower variance) across datasets.
Table 11: Paired 1-test p-values comparing our method against baselines. Bold indicates statistical significance (p 0.05).
Dataset
Method
Dice
ACC
AUC
SE
SP
DRIVE
HQ-SAM
9.86E-03
3.99E-04
2.47E-12
1.55E-01
1.32E-01
nnU-Net
5.50E-02
9.79E-06
2,44E-12
3.62E-03
2.08E-02
RETFound
1.38E-02
9.61E-07
1.38E-13
5.60E-03
5.96E-01
DCAI
HQ-SAM
4.53E-07
9.97E-01
1.07E-21
3.28E-06
2.57E-01
nnU-Net
8.79E-04
1.35E-14
2.10E-02
6.22E-03
1.14E-02
RETFound
1.24E-05
1.80E-05
4.79E-01
1.93-066
1.70E-02
CHUAC
HQ-SAM
1.04E-01
8.28E-08
1.71E-04
9.80E-03
6.41E-05
nnU-Net
6.96E-01
2.50E-01
2.50F-03
2.77E-01
4.46E-01
RETFouud
3,23E-01
7.69E-04
2.23E-01
3.12E-01
4.02E-01
ROSE
HQ-SAM 1.97E-03
3.20E-01
1.25E-01
1,4E-06
7.49E-01
nnU-Net 7.03E-01
3.13E-01
1.65E-01
3.02E-04
8.90E-01
RETFound
1.64E-04
1,51E-03
3.11E-01
6,43E-06
4.93E-02
Societal impact discussion
FineSAM+ is designed to mitigate localized failures in medical image segmentation patietttttioo-ppppilllllry in fine-grained regions (e.g.. vessels with blurred boundaries) vva structure- aware, sparsely activated expert mechanism that enhances fidelity with minimal computational overhead. This design advances two practical goals: (i) improving the reliability of automated tools that support clinical decision- making, thereby reducing the risk of missed spurious findings in delicate anatomical structures; and (ii) enabling a scalable, resource-efficient adaptation strategy that lowers deployment barriers in low-resource healthcare settings where retraining or maintaining large models is impractical.
G Limitations
While F.M+SAM++ demonstrates strong performance and robustness across multiple vascular segmen- tation benchmarks, several limitations remain. First, the current design employs a fixed number of Residual Experts with pre-defined perturbation settings, which may not fully capture the variability of structural errors across highly diverse anatomical regions. Future work could explore dynamic expert allocation or adaptive degradation strategies conditioned introduces additional complexity compared to single-expert or fully fine-tuned models. Third, our study focuses primarily on vascular datasets. Extension to other fine-grained medical segmentation tasks, such as tumor boundary refinement or organ delincation, remains to be validated. We believe these directions provide promising opportunities for further improving the generality and applicability of sparse expert refinement frameworks.
image content. Second, FineSAM+
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines: The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification: [TODO]
Guidelines: the paper discuss the limitations of the work performed.
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't "te't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, the paper provide the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: the paper fully disclose all the information needed to reproduce the main experimental results.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper uses publicly available data and provides sufficient explanations to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue htt.:///c//uiddee///eeeubbbb
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and testing details needed to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports defined error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For experiments, the paper provides information on the computer resources needed to reproduce the experiments.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper complies in all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses the possible social impact of the work carried out.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
23

---

Answer: [No]
Justification:
The paper uses public datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators original owners of assets used in the paper (e.g., code, data, models) are appropriately acknowledged, and the licenses and terms of use are clearly mentioned and appropriately respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpppsaiitcoooeeooosspp thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [No]
Justification:
The data used in the paper is public dataset.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [No]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: This study does involve any significant, original or non-standard LLM.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM for what should or should not be described.
25


Reviews:
[{"review_id": "qTCDMEhhHq", "reviewer": "Reviewer_6m4f", "summary": "This paper presents a structure-aware sparse expert framework, named FineSAM++, which enhances foundation segmentation models. The proposed method has a gating module that identifies structurally uncertain regions and activates a Residual Expert to correct local structural inconsistencies of vessel segmentation. The proposed method has been demonstrated on five vessel segmentation datasets, and the results show superior performance compared to the existing state-of-the-art segmentation methods.", "questions": "Please refer to the above weaknesses.\n\n- Is the proposed method efficient in terms of the number of learnable parameters among the SAM-based segmentation methods?\n- How is the error threshold (\\delta) in the gating module defined? Is the threshold the same for all five public datasets?\n- The hyperparameter study would strengthen the contribution of the proposed method.", "limitations": "yes (in supplementary material)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed framework with a gating module and residual expert refines the vessel segmentation mask from SAM model effectively.\n- The proposed method trains a part of the neural networks, which enables efficient training of the large model.\n- Experimental results using five public datasets demonstrate the effectiveness of the proposed method in local correction of the segmentation mask.\n\n### Weaknesses\n- There is no comparison for the size of learnable parameters between the proposed method and the existing SAM-based methods.\n- The proposed gating module requires a pre-defined error threshold (\\delta), which may make the model sensitive to the threshold value.\n- There is no study on the hyperparameters used in the gating module and residual experts, such as normalized routing weights (I, w_j) and the threshold (\\delta).", "was_revised": true, "final_justification": "After reviewing the rebuttal by the authors and other reviewers' comments, I keep my rating. The authors have addressed most of my concerns, and the SAM-based proposed method would be helpful in vessel segmentation masks of various medical images.", "rebuttal": "We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:\n\n**Q1&W1 “Parameter efficiency vs. SAM-based method”**:\n\nThanks for your valuable comment. To address your concern regarding the efficiency of our proposed method FineSAM++ in terms of learnable parameters among SAM-based segmentation methods, we have conducted a detailed comparison. Specifically, we computed the number of parameters, FLOPs (GAMCs), and inference latency under a standard input resolution of (1, 3, 1024, 1024). The results are summarized in the Tab.X below:\n\n**Table X.** Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).\n\n| **Model**     | **Params (M)**                | **GAMCs (G)** | **Latency (ms)** | **Dice**  |\n|---------------|-------------------------------|----------------|------------------|-----------|\n| Unets         | 34.53                         | 4.08           | 1.10             | 0.7787    |\n| nnUnet        | 126.2                         | 1864.9         | 37.4             | 0.8220    |\n| SAM Adapter   | 104.3                         | 400.1          | 127.8            | 0.4498    |\n| H-SAM         | 111.3                         | 370.6          | 124.8            | 0.6622    |\n| AutoSAM       | 135.29                        | 774.16         | 166.22           | 0.6603    |\n| SAMed         | 92.2                          | 370.5          | 117.1            | 0.6170    |\n| SAM           | 93.7                          | 372.0          | 116.33           | /         |\n| Ours (FineSAM++) | 94.4 | 376.8 | 117.6| 0.8231 |\n\nAs shown in the Tab.X, FineSAM++ introduces only 1.4M additional learnable parameters on top of the SAM backbone. Compared to other SAM-based segmentation methods—such as SAM Adapter (104.3M), H-SAM (111.3M), and AutoSAM (135.29M)—FineSAM++ demonstrates significantly higher parameter efficiency. Although the total parameter count is higher than lightweight models like Unet, FineSAM++ achieves the highest Dice score (0.8231). These results clearly indicate that FineSAM++ strikes an excellent balance between parameter efficiency and segmentation performance. We will include this comparison in the revised version.\n\n**Q2&W2 “Ablation study on threshold** $\\delta$ **in the Gating module”:**\n\nThank you for pointing this out. To assess the sensitivity of the Gating module to the pre-defined error threshold $\\delta$, we conducted an ablation study on the DRIVE dataset by varying $\\delta \\in \\{0.3, 0.4, 0.5, 0.6, 0.7\\}$. The results are summarized in the Tab.XI below:\n\n**Table XI**. The Ablation Result of threshold δ in the Gating module\n| **δ**      | **Dice** | **ACC** | **AUC** | **SE**        | **SP**        |\n|------------|----------|---------|---------|---------------|---------------|\n| 0.3        | 0.8124   | 0.9712  | 0.9812  | **0.8432**    | 0.9601        |\n| 0.4        | 0.8187   | 0.9755  | 0.9846  | 0.8410        | 0.9732        |\n| **0.5**    | **0.8231** | **0.9790** | **0.9870** | 0.8366        | **0.9834**    |\n| 0.6        | 0.8180   | 0.9767  | 0.9854  | 0.8204        | 0.9807        |\n| 0.7        | 0.8129   | 0.9735  | 0.9822  | 0.8083        | 0.9784        |\n\nAs shown, while some metrics (e.g., SE at $\\delta=0.3$) are slightly higher, $\\delta=0.5$ achieves the best overall performance across all metrics. Overall, $\\delta=0.5$ provides a meaningful routing threshold:  \n- If $\\delta$ is too low, almost all pixels are considered uncertain, resulting in unnecessary refinement and loss of gating sparsity.  \n- If $\\delta$ is too high, only a few pixels are routed, making the refinement module underutilized.  \n\nThus, $\\delta=0.5$ strikes a balance, activating residual experts in genuinely ambiguous regions while maintaining efficiency. Therefore, we adopt a fixed threshold $\\delta=0.5$ for all five public datasets for consistency. We will include this explanation and the ablation results in the final version.\n\n**Q3&W3 “There is no study on the hyperparameters used in the gating module and residual experts, such as normalized routing weights {w_j}. A hyperparameter study would strengthen the contribution”**:  \n\nThank you for the helpful suggestion. We address the reviewer's concerns as follows:\n\n**1.** Routing weights $\\{w_j\\}$ are dynamically learned.\n\nThe weights \n$\n\\lbrace w_j \\rbrace^J_{j=1}\n$\nare not manually defined hyperparameters. They are predicted dynamically by the Gating module $g_\\theta$, which outputs both the uncertainty mask $m \\in [0,1]^{H \\times W}$ and expert routing weights $\\{w_j\\}$, based on input $x$ and coarse prediction $\\hat{y}_{\\mathrm{SAM}}$.\n\n**2.** Softmax normalization ensures stability.\n\nTo enable robust and balanced routing, we apply softmax normalization to the predicted weights across experts at each spatial location $\\sum_{j=1}^{J} w_j = 1$. This prevents expert collapse and allows smooth specialization without hard routing decisions.\n\n**3.** Explanation of the indicator function $\\mathbb{I}(\\cdot)$.\n\nThe Gating module is trained using a binary supervision mask $g_t$ defined as:\n\n$$\ng_t(i) = \\mathbb{I}\\left(|\\hat{y}_{\\text{SAM}}(i) - y(i)| > \\delta\\right)\n$$\n\nwhere $\\mathbb{I}(\\cdot)$ denotes the standard indicator function, returning 1 if the condition holds and 0 otherwise. This provides supervision for pixels with high prediction error, helping the module identify uncertain regions.\n\n**4.** On the number of residual experts $J$. \n\nAs shown in Tab.3 of our manuscript, we performed an ablation study on $J \\in \\{1, 2, 4, 6\\}$. While $J=6$ achieved the highest accuracy, we chose $J=4$ in the final model to strike a better trade-off between segmentation performance and model complexity. This reduces the number of parameters and computational cost, while still maintaining strong performance."}, {"review_id": "YNUZF83Afx", "reviewer": "Reviewer_LmaJ", "summary": "The authors build on SAM to produce more coherent vessel segmentations. They introduce two modules: 1. A Routing module to identify uncertain regions. 2. A Residual Expert model (lightweight) that fixes the structures identified by the routing model.\nThey evaluate their method on 5 vessel datasets on many vessel/tube specific metrics including CLDice and connectivity.", "questions": "- Why did the authors choose different baselines per dataset ? It seems to weaken their evaluation and makes performance less comparable \n- What is the number of parameters with their suggested changes and how does it compare to other baselines ? (Especially SAM and UNets)\n- Why not mention nnUNets, and models specialized for tubular structures ?\n- Were the augmentations applied to all baselines as well? \n- Would it be possible to see standard deviations to assess the significance of the results?", "limitations": "The authors mentioned the added complexity and computation time of their method in the supplemental section. I think it would be great to quantify this numerically both in terms of parameters and in terms of training time (for both phases).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Strengths*\n- the authors evaluate on various datasets and modalities\n- a comprehensive set of metrics is used to assess both accuracy and topology of the produced segmentations. \n- the authors compare against a comprehensive set of baselines, including specialized models and UNets\n- the authors ran an ablation study to understand the significance of each of their module.\n\n*Weaknesses*.   \n\nSparse related work section.   \n- I would have expected more work on uncertainty evaluation, specialised models and foundation models. \n- No mention of the vessel specific literature. \n\nConfusion in the experiments. \n\n- Different baselines are used for different datasets. Why is this the case? Can the author elaborate? I would guess that this limits the strength of the paper. \n- Why are the UNet results so variable depending on the dataset? \n- Did you also run data augmentations for the baselines? \nIt seems to me that some important baselines are missing: \n- SAM models specifically designed for fine structures such as _Segment anything in high quality_ [1]\n- nnUNet, which are a standard network for medical image segmentation [2]\n- Foundation model for tubular structures   \n\n[1]: Ke, Lei, et al. \"Segment anything in high quality.\" Advances in Neural Information Processing Systems 36 (2023): 29914-29934.  \n[2]: Isensee, Fabian, et al. \"nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.\" Nature methods 18.2 (2021): 203-211.", "was_revised": true, "final_justification": "I appreciate the authors effort to address all my concerns. The marginal improvements on nnUNet seem a bit concerning (sometimes are insignificance). If the main claim is latency, this should be more emphasized in the paper. I think the improvements in latency are interesting though, but should be clarified in the paper.", "rebuttal": "We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:\n\n**W1 “Sparse related work section”:**\n\nWe appreciate the reviewer’s observation regarding the breadth of the related work. We would like to clarify that our Related Work section (Sec. 2) will include dedicated subsections that cover foundation models, uncertainty modeling, specialized approaches, and vessel-specific segmentation literature: \n\n**1)Foundation Models & Uncertainty modeling**: We will expand Section 2.1 to cover additional efforts on adapting foundation models for medical image segmentation, including LoRA-based fine-tuning [1], prompt-driven adaptation [2], and SAM-Adapter methods [3]. To address the reviewer’s interest in uncertainty and boundary quality, we will add HQ-SAM [4], which improves boundary precision via high-quality priors and edge-aware refinement. We will also discuss topology-aware uncertainty modeling [5,6], which enhances consistency by quantifying structural uncertainty. These additions better contextualize the potential and limitations of foundation models in segmenting fine-grained medical structures like vessels.\n\n**2) Specialized Models for Vessel Segmentation**: We will expand Section 2.3 to include recent progress in both traditional and foundation model-based vessel segmentation. Beyond CNN and Transformer methods [7,8], we will add vesselFM [9], a foundation model for universal 3D vessel segmentation, and a retinal disease foundation model [10], which supports generalizable vessel-pathology detection. These works underscore the increasing relevance of foundation models in vascular imaging for both segmentation and clinical insight.\n\n**Q1, 3, 4, 5 & W2.1, 2.3, 2.4, 2.5, 2.6 “Baseline selection, data augmentations, missing import baselines and standard deviations”:**\n\nWe thank the reviewer for raising this important point. For each dataset, we selected baselines that are both highly cited on Google Scholar and representative of state-of-the-art performance, aiming to reflect the most influential methods specific to each task. Importantly, we included several SAM-based methods (SAM-Adapter, H-SAM, AutoSAM, SAMed) across all datasets, along with a vessel-specific baseline (Gupta et al. [5]). We agree that a unified suite of baselines would improve consistency, and we will revise our experiments to add some important baselines:\n\n1)\tHQ-SAM [4]: A SAM variant specifically designed for fine structural segmentation. \n2)\tnnUNet [11]: A standard network for medical image segmentation. \n3)\tRETFound (modified) [10]: A foundation model for vessel disease detection. We adapt it for segmentation by replacing the classification head with a lightweight decoder.\n\nFurthermore, to ensure fairness, all methods were trained with the same data augmentation strategies, following [5], which includes random rotation, flipping, elastic deformation, and brightness/contrast jittering. \n\nAs shown in Table XII and Table 1, HQ-SAM improves over other SAM-based models (e.g., H-SAM, SAM-Adapter, SAMed) via lightweight token-level refinement. However, its frozen encoder and global adaptation lack domain-specific priors and struggle with fine-grained structures such as thin vessels and bifurcations. RETFound performs better structurally due to pretraining, while nnUNet remains stable due to its self-configuring pipeline and strong inductive bias in medical segmentation. Our method achieves balanced and robust performance across datasets, especially on Dice and AUC, critical for fine vessel structures. A more detailed analysis will be given in the revised version.\n\nTo better evaluate the robustness of our method, we report the mean ± standard deviation in Table XII. Minor revisions will be made in the final version to include these statistics for completeness.\n\n**Table XII.** Quantitative comparison on DRIVE, DCAI, CHUAC and ROSE datasets.\n| **Dataset** | **Method**    | **Dice**           | **ACC**            | **AUC**            | **SE**             | **SP**             |\n|---------|-----------|----------------|----------------|----------------|----------------|----------------|\n| DRIVE   | HQ-SAM    | 0.7978         | 0.9697         | 0.8824         | 0.8033         | 0.9824         |\n|         | nnUnet    | 0.822          | 0.9698         | 0.894          | 0.8019         | 0.9862         |\n|         | RETFound  | 0.802          | 0.9649         | 0.883          | 0.7796         | 0.9821         |\n|         | Ours      | **0.8231±0.014** | **0.979±0.003** | **0.987±0.023** | **0.8366±0.048** | **0.9834±0.004** |\n| DCAI    | HQ-SAM    | 0.788          | 0.977          | 0.889          | 0.789          | 0.988          |\n|         | nnUnet    | 0.8045         | 0.9584         | 0.9903         | 0.8264         | 0.9879         |\n|         | RETFound  | 0.7938         | 0.9681         | 0.9911         | 0.8853         | 0.9861         |\n|         | Ours      | **0.8127**     | **0.9775**     | **0.9931**     | **0.8479**     | **0.9872**     |\n| CHUAC   | HQ-SAM    | 0.705          | 0.894          | 0.812          | 0.676          | 0.948          |\n|         | nnUnet    | 0.7814         | 0.9776         | 0.8842         | 0.7788         | 0.9896         |\n|         | RETFound  | 0.7636         | 0.9604         | 0.9904         | 0.7325         | 0.9906         |\n|         | Ours      | **0.7768±0.045** | **0.9807±0.005** | **0.9951±0.032** | **0.7567±0.065** | **0.9932±0.003** |\n| ROSE    | HQ-SAM    | 0.752          | 0.9609         | 0.9904         | 0.794          | 0.9887         |\n|         | nnUnet    | 0.827          | 0.947          | 0.931          | 0.865          | 0.994          |\n|         | RETFound  | 0.7126         | 0.9197         | 0.9337         | 0.8563         | 0.9193         |\n|         | Ours      | **0.822±0.047** | **0.9483±0.027** | **0.9827±0.046** | **0.9485±0.047** | **0.9823±0.094** |\n\n**W2.2 “Why are the UNet results so variable depending on the dataset?”:**\n\nThank you for the question. The performance variability of UNet reflects both dataset characteristics and inherent model limitations. UNet relies heavily on low-level intensity cues and lacks global context modeling, making it sensitive to modality shifts and low-contrast scenarios. For instance, it performs well on DRIVE (high-contrast RGB) but struggles on CHUAC (grayscale, noisy angiography). This highlights UNet’s limited robustness across diverse medical domains.\n\n**Q2 & L1 “Parameter Count Comparison”:**\n\nWe thank the reviewers for their comments. To better illustrate the computational properties of our method, we compare its performance against SAM-based methods, UNets, and nnUNet. While training time is a relevant consideration, it is not the primary bottleneck or design objective of our approach. FineSAM++ is built upon a frozen backbone and introduces two lightweight adaptation stages, both of which converge efficiently in practice. Accordingly, we focus on parameter efficiency, inference cost, and segmentation accuracy. The results are summarized in Table X below:\n\n**Table X.** Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).\n\n| **Model**     | **Params (M)**                | **GAMCs (G)** | **Latency (ms)** | **Dice**  |\n|---------------|-------------------------------|----------------|------------------|-----------|\n| Unets         | 34.53                         | 4.08           | 1.10             | 0.7787    |\n| nnUnet        | 126.2                         | 1864.9         | 37.4             | 0.8220    |\n| SAM Adapter   | 104.3                         | 400.1          | 127.8            | 0.4498    |\n| H-SAM         | 111.3                         | 370.6          | 124.8            | 0.6622    |\n| AutoSAM       | 135.29                        | 774.16         | 166.22           | 0.6603    |\n| SAMed         | 92.2                          | 370.5          | 117.1            | 0.6170    |\n| SAM           | 93.7                          | 372.0          | 116.33           | /         |\n| Ours (FineSAM++) | 94.4 | 376.8 | 117.6| 0.8231 |\n\nAs shown above, FineSAM++ achieves a Dice score of 0.8231, the highest among all SAM-based methods, while introducing only 1.4M additional learnable parameters. Compared to UNet (1.1 ms latency) and SAM (116.33 ms latency), our method maintains a comparable inference time (117.6 ms) and significantly improves performance over Unet. In contrast to nnUNet, which requires heavy computation (1864.9G FLOPs), FineSAM++ achieves similar accuracy (0.8231 vs. 0.8220) with over 4× fewer parameters and over 30× faster inference.\n\nReferences:\n\n[1] Cheng Z., et al (2024). Unleashing the potential of sam for medical adaptation via hierarchical decoding. \n\n[2] Chen Z., et al (2025). UN-SAM: Domain-adaptive self-prompt segmentation for universal nuclei images.\n\n[3] Wu J., et al (2025). Medical sam adapter: Adapting segment anything model for medical image segmentation.\n\n[4] Ke L., et al (2023). Segment anything in high quality. \n\n[5] Gupta S., et al (2023). Topology-aware uncertainty for image segmentation.\n\n[6] Huang J., et al (2024). Representing topological self-similarity using fractal feature maps for accurate segmentation of tubular structures.\n\n[7] Qi Y., et al (2023). Dynamic snake convolution based on topological geometric constraints for tubular structure segmentation.\n\n[8] Mou L., et al (2024). CS-Net: Channel and spatial attention network for curvilinear structure segmentation.\n\n[9] Wittmann B., et al (2025). vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation.\n\n[10] Zhou Y., et al (2023). A foundation model for generalizable disease detection from retinal images.\n\n[11] Isensee F., et al (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation."}, {"review_id": "JQfqGxqEHJ", "reviewer": "Reviewer_gHFg", "summary": "This paper introduces FineSAM++, a structure-aware sparse expert framework designed to enhance the performance of foundation segmentation models like SAM on medical images, particularly for fine-grained structures such as vessels. FineSAM++ employs a confidence-driven soft Routing Module to dynamically identify uncertain regions and selectively activate a lightweight Residual Expert for local refinement, enabling efficient correction without full retraining.", "questions": "See above", "limitations": "It is recommended to provide a discussion of the societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The idea of proposed method is simple and straightforward.\n2. The proposed method achieves state-of-the-art performance on five publicly available vascular segmentation datasets.\n\nWeaknesses:\n1. While SAM demonstrates strong generalization across various diseases and imaging modalities, it would be helpful to clarify why the proposed MoE framework is applied only to vessel segmentation. Extending the approach to other structures or modalities could further validate its generalizability.\n2. It is recommended to provide a comparison of the number of trainable parameters and the inference FLOPs between FineSAM++ and baseline methods, to better illustrate the computational efficiency of the proposed framework.\n3. The paper would benefit from additional ablation studies or analysis on hyperparameters, such as the impact of $\\delta$ in Equation 3, to help understand their influence on performance.\n4. A more detailed description of the method, including the design of $g_\\theta$  in Equation 2 and the architecture of the expert modules, would improve the clarity and reproducibility of the work.\n5. It is suggested to evaluate the effectiveness of FineSAM++ on multi-class segmentation and different disease segmentation tasks to further demonstrate its versatility and robustness.", "was_revised": true, "final_justification": "As the authors have addressed some of the previous concerns, I have increased my score for acceptance.", "rebuttal": "We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:\n\n**W1, 5 “Extending the approach to other structures or modalities to validate its generalizability.”:**\n\nWe agree with the concern regarding the importance of validating the generalizability of our proposed framework beyond vessel segmentation. In response, we have extended our evaluation to multi-class segmentation on the Synapse Multi-Organ CT dataset, which includes eight anatomically diverse abdominal organs. Following [1, 8, 9], we split the dataset into 18 training samples and 12 test samples, and perform corresponding preprocessing and data augmentation methods.\n\nWe present the quantitative results in Tab.XII. As shown, our method achieves the highest average Dice score (87.97%) and the lowest Hausdorff Distance (HD) of 7.89 among all compared methods, outperforming strong baselines such as H-SAM and nnU-Net. Notably, our method maintains high segmentation accuracy on challenging small organs like the pancreas, demonstrating that FineSAM++ preserves its segmentation strength and robustness even when extended to more delicate anatomical structures. We will include this supplementary analysis in the final submission to better showcase the generalization capability of our approach and further support the robustness of the method.\n\n**Table XI.** Comparison to state-of-the-art models on Synapse multi-organ CT dataset.\n\n| Method        | Spleen | Right Kidney | Left Kidney | Gallbladder | Liver | Stomach | Aorta | Pancreas | Mean Dice (%) | HD    |\n|---------------|--------|--------------|-------------|-------------|-------|---------|-------|----------|----------------|--------|\n| TransUnet [1]     | 87.23  | 63.13        | 81.87       | 77.02       | 94.08 | 55.86   | 85.08 | 75.62    | 77.48          | 31.69  |\n| SwinUnet [2]     | 85.47  | 66.53        | 83.28       | 79.61       | 94.29 | 56.58   | 90.66 | 76.6     | 79.13          | 21.55  |\n| TransDeepLab [3] | 86.04  | 69.16        | 84.08       | 79.88       | 93.53 | 61.19   | 89.00 | 78.4     | 80.16          | 21.25  |\n| DAE-Former [4]   | 88.96  | 72.3         | 86.08       | 80.88       | 94.98 | 65.12   | 91.94 | 79.19    | 82.43          | 17.46  |\n| MERIT [5]        | 92.01  | 84.85        | 87.79       | 74.4        | 95.26 | 85.38   | 87.71 | 71.81    | 84.9           | 13.22  |\n| nnUnet [10]       | 91.68  | 88.46        | 83.68       | 70.82       | 97.13 | 83.34   | 93.04 | 81.5     | 87.33          | 10.78  |\n| AutoSAM [6]      | 80.54  | 80.02        | 79.6        | 41.37       | 89.24 | 61.14   | 82.56 | 44.22    | 62.08          | 27.56  |\n| SAM Adapter [7]  | 83.68  | 79.00        | 79.02       | 57.49       | 92.67 | 69.48   | 77.93 | 43.07    | 72.8           | 33.08  |\n| SAMed [8]        | 87.77  | 69.11        | 80.45       | 79.95       | 94.8  | 72.17   | 88.72 | 82.06    | 81.88          | 20.64  |\n| H-SAM [9]        | 93.34  | 89.93        | 91.88       | 73.49       | 95.72 | 87.1    | 89.38 | 71.11    | 86.49          | 8.18   |\n| **Ours**      | **94.25** | **91.53**    | **93.21**   | **71.23**   | **96.89** | **90.83** | **92.52** | **82.23**  | **87.97**        | **7.89** |\n\n**W2 “Parameter efficiency”:**\n\nThanks for your valuable comment. To address your concern regarding the efficiency of our proposed method FineSAM++ in terms of learnable parameters among SAM-based segmentation methods, we have conducted a detailed comparison. Specifically, we computed the number of parameters, FLOPs (GAMCs), and inference latency under a standard input resolution of (1, 3, 1024, 1024). The results are summarized in the Tab.X below:\n\n**Table X.** Comparison of model size, computational cost, latency, and segmentation accuracy (Dice score) across segmentation methods using an input of size (1, 3, 1024, 1024).\n\n| **Model**     | **Params (M)**                | **GAMCs (G)** | **Latency (ms)** | **Dice**  |\n|---------------|-------------------------------|----------------|------------------|-----------|\n| Unets         | 34.53                         | 4.08           | 1.10             | 0.7787    |\n| nnUnet        | 126.2                         | 1864.9         | 37.4             | 0.8220    |\n| SAM Adapter   | 104.3                         | 400.1          | 127.8            | 0.4498    |\n| H-SAM         | 111.3                         | 370.6          | 124.8            | 0.6622    |\n| AutoSAM       | 135.29                        | 774.16         | 166.22           | 0.6603    |\n| SAMed         | 92.2                          | 370.5          | 117.1            | 0.6170    |\n| SAM           | 93.7                          | 372.0          | 116.33           | /         |\n| Ours (FineSAM++) | 94.4 | 376.8 | 117.6| 0.8231 |\n\nAs shown in the Tab.X, FineSAM++ introduces only 1.4M additional learnable parameters on top of the SAM backbone. Compared to other SAM-based segmentation methods, FineSAM++ demonstrates significantly higher parameter efficiency. Although the total parameter count is higher than lightweight models like Unet, FineSAM++ achieves the highest Dice score (0.8231). These results clearly indicate that FineSAM++ strikes an excellent balance between parameter efficiency and segmentation performance. We will include this comparison in the revised version.\n\n**W3 “Ablation study on threshold** $\\delta$ **in the Gating module”:**\n\nThanks for pointing this out. To assess the sensitivity of the Gating module to the pre-defined error threshold $\\delta$, we conducted an ablation study on the DRIVE dataset by varying $\\delta \\in \\{0.3, 0.4, 0.5, 0.6, 0.7\\}$. The results are summarized in the Tab.XI below:\n\n**Table XI**. The Ablation Result of threshold δ in the Gating module\n| **δ**      | **Dice** | **ACC** | **AUC** | **SE**        | **SP**        |\n|------------|----------|---------|---------|---------------|---------------|\n| 0.3        | 0.8124   | 0.9712  | 0.9812  | **0.8432**    | 0.9601        |\n| 0.4        | 0.8187   | 0.9755  | 0.9846  | 0.8410        | 0.9732        |\n| **0.5**    | **0.8231** | **0.9790** | **0.9870** | 0.8366        | **0.9834**    |\n| 0.6        | 0.8180   | 0.9767  | 0.9854  | 0.8204        | 0.9807        |\n| 0.7        | 0.8129   | 0.9735  | 0.9822  | 0.8083        | 0.9784        |\n\nAs shown, while some metrics are slightly higher, $\\delta=0.5$ achieves the best overall performance across all metrics. Overall, $\\delta=0.5$ provides a meaningful routing threshold:\n  \n- If $\\delta$ is too low, almost all pixels are considered uncertain, resulting in unnecessary refinement and loss of gating sparsity.  \n- If $\\delta$ is too high, only a few pixels are routed, making the refinement module underutilized.\n  \nThus, $\\delta=0.5$ strikes a balance, activating residual experts in genuinely ambiguous regions while maintaining efficiency. We will include this explanation and the ablation results in the final version.\n\n**W4 “A more detailed description of the method, including the design of $g_\\theta$ in Equation 2 and the architecture of the expert modules”:**\n\nWe thank the reviewer for emphasizing the importance of methodological clarity. We will revise the manuscript to provide a more detailed description.\n\n**1.** Gating Module $g_\\theta$ (Eq. 2):\n\n$g_\\theta$ is implemented as a lightweight CNN without downsampling. It contains three convolutional layers with kernel size $3 \\times 3$, followed by BatchNorm and ReLU activation. It produces two outputs through separate heads:\n\n•\tUncertainty map: A $1 \\times 1$ convolution followed by a sigmoid activation outputs a spatial confidence map.\n\n•\tRouting weights: The feature map is globally pooled and passed through an MLP to produce per-image expert routing weights.\n\n**2.** Expert Modules:\n\nEach expert is a compact 3-level U-Net, where each level contains 2 down-sampling and 2 up-sampling blocks. All experts share architecture but are independently trained to specialize in different error patterns. The final prediction is a weighted sum of expert outputs, gated by the spatial uncertainty and routing weights. We will update the final paper and appendix to support reproducibility.\n\n**L1 “Societal impact discussion”:**\n\nWe appreciate the reviewer’s suggestion to address societal impact. FineSAM++ targets localized failures in medical images, especially in fine-grained regions like vessels with blurred boundaries, by introducing a structure-aware, sparsely activated expert mechanism that improves segmentation fidelity with minimal overhead. This design offers two key benefits: (1) improved reliability of automated tools to support clinical decision-making, and (2) a scalable, resource-efficient adaptation strategy that facilitates deployment in low-resource healthcare settings where retraining large models is impractical.\n\nReferences\n\n[1] Chen J., et al (2021). Transunet: Transformers make strong encoders for medical image segmentation.\n\n[2] Cao H., et al (2022). Swin-unet: Unet-like pure transformer for medical image segmentation.\n\n[3] Azad R., et al (2022). Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation.\n\n[4] Azad R., et al (2023). Dae-former: Dual attention-guided efficient transformer for medical image segmentation.\n\n[5] Rahman M M., et al (2024). Multi-scale hierarchical vision transformer with cascaded attention decoding for medical image segmentation.\n\n[6] Hu X., et al (2023). How to efficiently adapt large segmentation model (sam) to medical images.\n\n[7] Chen T., et al (2023). Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more.\n\n[8] Zhang K., et al (2023). Customized segment anything model for medical image segmentation.\n\n[9] Cheng Z., et al (2024). Unleashing the potential of sam for medical adaptation via hierarchical decoding.\n\n[10] Isensee F., et al (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation."}, {"review_id": "H0hM217AC8", "reviewer": "Reviewer_ospL", "summary": "Segmentation foundation models like SAM demonstrate strong generalization on natural images but tend to underperform on medical images, which often involve fine-grained structures such as blood vessels. This paper introduces FineSAM++, a sparse expert framework designed to refine SAM's output. Specifically, it employs a confidence-driven soft routing module to identify uncertain regions and selectively activate mixed-of-residual-experts to correct structural errors.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe proposed framework is well-motivated. While SAM is powerful, it struggles to capture fine-grained, high-resolution details—especially in medical images. Instead of retraining the entire model, this work proposes a more efficient approach: obtain an initial coarse mask, identify uncertain regions, and apply residual corrections through specialized experts. These diverse Residual Experts are trained to address different types of errors, such as disconnections, missing thin vessels, or noisy boundaries.\n\nThe paper includes thorough ablation studies that demonstrate:\n\n- the effectiveness of using multiple specialized Residual Experts over a single expert, and\n\n- the benefit of the proposed gating module for selective refinement.\n\nA comprehensive evaluation is conducted across multiple datasets with a wide range of baselines, showing that the proposed method delivers competitive performance.\n\nWeaknesses:\n\nWhile it is intuitive to perform refinement on top of coarse masks from a foundation model, it would be important to acknowledge and analyze potential failure cases. In my own experience, refinement modules can sometimes overfit to thin structures, leading to false positives and degrading the initial mask quality. Did the authors observe any such failure modes where refinement worsened the output?\n\nThe paper lacks an ablation on the progressive optimization strategy with dynamic weighting. It would be helpful to compare this approach with a simpler stage-wise training setup, where each correction module is trained independently. A discussion on the specific advantages of the proposed strategy is needed.\n\nThe Residual Experts appear to be relatively independent from the base model. Have the authors explored lightweight, orthogonal integration methods such as adapters? This could offer insights into whether the refinement process is generalizable.", "was_revised": true, "final_justification": "The response from the author solves most of my concerns. I will maintain my initial recommendation on accepting this paper", "rebuttal": "We thank reviewer for his/her valuable and insightful reviews. Here we address his/her main concerns:\n\n**W1 “While it is intuitive to perform refinement on top of coarse masks from a foundation model, it would be important to acknowledge and analyze potential failure cases. In my own experience, refinement modules can sometimes overfit to thin structures, leading to false positives and degrading the initial mask quality. Did the authors observe any such failure modes where refinement worsened the output?”:**\n\nWe thank the reviewer for the insightful comment. We agree that refinement modules can sometimes overfit to thin structures, potentially leading to false positives or degraded mask quality—especially in regions with weak boundaries or ambiguous textures. This is an important and realistic concern that we have also observed in our experiments.\n\nTo address this, we adopt a Degrade strategy in FineSAM++. Instead of feeding all experts the same coarse mask, we introduce randomized degradations:\n\n$$\n\\hat{y}_{\\text{SAM}}^{(j)}=\\text{Degrade} ( {\\hat y}\\_{SAM}, \\eta\\_j )\n$$\n\nwhere $\\text{Degrade}(\\cdot)$ applies random masking, noise injection, or occlusion perturbations to promote coarse mask input diversity. This encourages specialization across experts and improves robustness to structural uncertainty. We conducted an ablation experiment to evaluate the effectiveness of the degradation mechanism. The results are summarized in Table XV.\n\n**Table XV.** Ablation study of the Degrade strategy for multi-expert training.\n| Strategy      | Dice    | ACC    | AUC    | SE     | SP     |\n|---------------|---------|--------|--------|--------|--------|\n| No Degrade  | 0.8154  | 0.9735 | 0.9813 | 0.8281 | 0.9720 |\n| Degrade    | **0.8231** | **0.9790** | **0.9870** | 0.8366 | **0.9834** |\n\nThe Degrade strategy consistently improved robustness, reduced false positives, and encouraged specialization across experts. We will include these results and visual examples in the revised submission and supplemental material.\n\n**W2: The ablation of progressive optimization strategy**\n\nThank you for the insightful suggestion. We agree that analyzing the effect of the progressive optimization strategy with dynamic weighting is important. To this end, we compared three strategies: \n\n1）Naïve Joint Training: end-to-end optimization with uniform loss weights.\n\n2）Stage-wise (Independent): freeze coarse module, train refinement separately. \n\n3）Ours (Progressive): progressive training with dynamic uncertainty-based loss weighting. \n\nThe results are summarized in Table XVI. Our method consistently outperforms both baselines. The progressive setup encourages interaction between modules and focuses learning on uncertain regions, which improves refinement without overfitting. Our method consistently outperforms both baselines. The progressive setup encourages interaction between modules and focuses learning on uncertain regions, which improves refinement without overfitting. We will include these results and discussion in the revised paper. Thank you again for the helpful suggestion. We will include these results and discussion in the revised paper. Thank you again for the helpful suggestion.\n\n**Table. XVI**: Ablation study on training strategies.\n| Strategy                | Dice   | ACC    | AUC    | SE     | SP     |\n|-------------------------|--------|--------|--------|--------|--------|\n| Naïve Joint Training    | 0.7984 | 0.9641 | 0.9745 | 0.8123 | 0.9632 |\n| Stage-wise (Independent)| 0.8117 | 0.9722 | 0.9810 | 0.8289 | 0.9766 |\n| **Ours (Progressive)**  | **0.8231** | **0.9790** | **0.9870** | **0.8366** | **0.9834** |\n\n**W3 “The Residual Experts appear to be relatively independent from the base model. Have the authors explored lightweight, orthogonal integration methods such as adapters? This could offer insights into whether the refinement process is generalizable”:**\n\nWe thank the reviewer for the insightful comment. We have indeed explored adapter-based refinement as a baseline, inspired by recent methods such as SAM-Adapter [1]. While adapters provide a parameter-efficient integration mechanism, we found them suboptimal for our task: they struggle to correct fine-grained structures such as thin vessels, bifurcations, and topological continuity.\n\nAs shown in Tab. L and Fig. 4 (in manuscript), adapter-based methods often fail to recover fragmented branches or subtle boundaries, especially in high-precision vessel segmentation. These limitations motivated us to explore a mechanism that activates refinement paths only when needed—allowing the model to focus capacity on challenging regions while maintaining overall efficiency.\n\n**Table. X**: Comparison with SAM-Adapter across vessel datasets.\n\n| Dataset | Methods      | **Dice** | **ACC**  | **AUC**  | **SE**   | **SP**   | **C**    | **A**    | **L**    | **ClDice** |\n|---------|--------------|----------|----------|----------|----------|----------|----------|----------|----------|------------|\n| Drive   | SAM-Adapter  | 0.4498   | 0.9311   | 0.9204   | 0.7577   | 0.9377   | 0.993    | 0.412    | 0.321    | 0.488      |\n|         | **Ours**     | 0.8231   | 0.9790   | 0.9870   | 0.8366   | 0.9834   | 0.998    | 0.848    | 0.865    | 0.832      |\n| DCA1    | SAM-Adapter  | 0.7583   | 0.9727   | 0.9408   | 0.7882   | 0.9836   | 0.998    | 0.785    | 0.812    | 0.800      |\n|         | **Ours**     | 0.8127   | 0.9775   | 0.9931   | 0.8479   | 0.9872   | 0.997    | 0.903    | 0.877    | 0.865      |\n| CHUAC   | SAM-Adapter  | 0.7636   | 0.9784   | 0.9359   | 0.7583   | 0.9902   | 0.998    | 0.759    | 0.768    | 0.750      |\n|         | **Ours**     | 0.7768   | 0.9807   | 0.9951   | 0.7567   | 0.9932   | 0.998    | 0.787    | 0.795    | 0.770      |\n| ROSE    | SAM-Adapter  | 0.6316   | 0.8578   | 0.8451   | 0.6503   | 0.9801   | 0.992    | 0.683    | 0.657    | 0.660      |\n|         | **Ours**     | 0.8220   | 0.9483   | 0.9827   | 0.9485   | 0.9823   | 0.997    | 0.819    | 0.839    | 0.805      |\n\nMixture-of-Experts (MoE) architectures are well-suited for modeling spatially sparse and heterogeneous error patterns [2–4], which are characteristic of SAM’s failure cases in medical imaging. Our confidence-guided routing module enables spatially adaptive specialization by activating lightweight Residual Experts only in uncertain regions. Unlike bottlenecked adapter layers, these CNN-based experts retain local structural priors critical for topology-aware refinement.\n\nMoreover, while SAM-Adapter [1] focuses on global adaptation in natural scenes (e.g., camouflage, shadows), our method is designed for localized, structure-aware correction—essential in clinical settings where boundary precision and topological integrity are crucial. FineSAM++ consistently outperforms both adapter-based and task-specific baselines across five vessel benchmarks, demonstrating the effectiveness and generalizability of this sparse, modular correction paradigm.\n\nReferences\n\n[1] Chen T., et al (2023). Sam-adapter: Adapting segment anything in underperformed scenes.\n\n[2] Jacobs R A., et al (1991). Adaptive mixtures of local experts[J]. Neural computation.\n\n[3] Fedus W., et al (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.\n\n[4] Hwang C., et al (2023). Tutel: Adaptive mixture-of-experts at scale."}]

Author rebuttal:

Meta-review: The paper introduces FineSAM++, a structure-aware, sparse Mixture-of-Experts add-on for SAM that routes only uncertain, topology-critical regions (e.g., fine vessels) to lightweight Residual Experts for refinement, leaving confident regions untouched. The paper demonstrates a clear, substantive improvement on an important application with a simple, interpretable mechanism that is broadly useful for adapting foundation models in clinical settings. All the reviewers gave positive ratings. Therefore, meta reviewer agrees with the reviewers for the final recommendation.
Decision: Accept (poster)