Title: Gate to the Vessel: Residual Experts Restore What SAM Overlooks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Medical Image Segmentation ; Foundation Models;  Residual Learning;  Sparse Expert Modules
Abstract: Foundation segmentation models like Segment Anything (SAM) exhibit strong generalization on natural images but struggle with localized failures in medical imaging, especially on fine-grained structures such as vessels with complex morphology and indistinct boundaries. To address this, we propose FineSAM++, a structure-aware sparse expert framework designed to refine SAM outputs by introducing a confidence-driven soft Routing Module. This module dynamically identifies structurally uncertain regions and activates a lightweight Residual Expert to model and correct residual structural errors only within these areas, thereby achieving efficient "refinement over retraining." Extensive experiments on five public vascular segmentation datasets demonstrate that FineSAM++ consistently outperforms both SAM-adapted baselines and task-specific models in terms of accuracy, topological consistency. Our results highlight the effectiveness of sparse, structure-driven Mixture-of-Experts (MoE) strategies for enhancing the reliability of foundation vision models in clinical image understanding tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSegmentation foundation models like SAM demonstrate strong generalization on natural images but tend to underperform on medical images, which often involve fine-grained structures such as blood vessels. This paper introduces FineSAM++, a sparse expert framework designed to refine SAM's output. Specifically, it employs a confidence-driven soft routing module to identify uncertain regions and selectively activate mixed-of-residual-experts to correct structural errors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe proposed framework is well-motivated. While SAM is powerful, it struggles to capture fine-grained, high-resolution details—especially in medical images. Instead of retraining the entire model, this work proposes a more efficient approach: obtain an initial coarse mask, identify uncertain regions, and apply residual corrections through specialized experts. These diverse Residual Experts are trained to address different types of errors, such as disconnections, missing thin vessels, or noisy boundaries.\n\nThe paper includes thorough ablation studies that demonstrate:\n\n- the effectiveness of using multiple specialized Residual Experts over a single expert, and\n\n- the benefit of the proposed gating module for selective refinement.\n\nA comprehensive evaluation is conducted across multiple datasets with a wide range of baselines, showing that the proposed method delivers competitive performance.\n\nWeaknesses:\n\nWhile it is intuitive to perform refinement on top of coarse masks from a foundation model, it would be important to acknowledge and analyze potential failure cases. In my own experience, refinement modules can sometimes overfit to thin structures, leading to false positives and degrading the initial mask quality. Did the authors observe any such failure modes where refinement worsened the output?\n\nThe paper lacks an ablation on the progressive optimization strategy with dynamic weighting. It would be helpful to compare this approach with a simpler stage-wise training setup, where each correction module is trained independently. A discussion on the specific advantages of the proposed strategy is needed.\n\nThe Residual Experts appear to be relatively independent from the base model. Have the authors explored lightweight, orthogonal integration methods such as adapters? This could offer insights into whether the refinement process is generalizable.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FineSAM++, a structure-aware sparse expert framework designed to enhance the performance of foundation segmentation models like SAM on medical images, particularly for fine-grained structures such as vessels. FineSAM++ employs a confidence-driven soft Routing Module to dynamically identify uncertain regions and selectively activate a lightweight Residual Expert for local refinement, enabling efficient correction without full retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of proposed method is simple and straightforward.\n2. The proposed method achieves state-of-the-art performance on five publicly available vascular segmentation datasets.\n\nWeaknesses:\n1. While SAM demonstrates strong generalization across various diseases and imaging modalities, it would be helpful to clarify why the proposed MoE framework is applied only to vessel segmentation. Extending the approach to other structures or modalities could further validate its generalizability.\n2. It is recommended to provide a comparison of the number of trainable parameters and the inference FLOPs between FineSAM++ and baseline methods, to better illustrate the computational efficiency of the proposed framework.\n3. The paper would benefit from additional ablation studies or analysis on hyperparameters, such as the impact of $\\delta$ in Equation 3, to help understand their influence on performance.\n4. A more detailed description of the method, including the design of $g_\\theta$  in Equation 2 and the architecture of the expert modules, would improve the clarity and reproducibility of the work.\n5. It is suggested to evaluate the effectiveness of FineSAM++ on multi-class segmentation and different disease segmentation tasks to further demonstrate its versatility and robustness.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nIt is recommended to provide a discussion of the societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors build on SAM to produce more coherent vessel segmentations. They introduce two modules: 1. A Routing module to identify uncertain regions. 2. A Residual Expert model (lightweight) that fixes the structures identified by the routing model.\nThey evaluate their method on 5 vessel datasets on many vessel/tube specific metrics including CLDice and connectivity.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- the authors evaluate on various datasets and modalities\n- a comprehensive set of metrics is used to assess both accuracy and topology of the produced segmentations. \n- the authors compare against a comprehensive set of baselines, including specialized models and UNets\n- the authors ran an ablation study to understand the significance of each of their module.\n\n*Weaknesses*.   \n\nSparse related work section.   \n- I would have expected more work on uncertainty evaluation, specialised models and foundation models. \n- No mention of the vessel specific literature. \n\nConfusion in the experiments. \n\n- Different baselines are used for different datasets. Why is this the case? Can the author elaborate? I would guess that this limits the strength of the paper. \n- Why are the UNet results so variable depending on the dataset? \n- Did you also run data augmentations for the baselines? \nIt seems to me that some important baselines are missing: \n- SAM models specifically designed for fine structures such as _Segment anything in high quality_ [1]\n- nnUNet, which are a standard network for medical image segmentation [2]\n- Foundation model for tubular structures   \n\n[1]: Ke, Lei, et al. \"Segment anything in high quality.\" Advances in Neural Information Processing Systems 36 (2023): 29914-29934.  \n[2]: Isensee, Fabian, et al. \"nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.\" Nature methods 18.2 (2021): 203-211.\n\nQUESTIONS:\n- Why did the authors choose different baselines per dataset ? It seems to weaken their evaluation and makes performance less comparable \n- What is the number of parameters with their suggested changes and how does it compare to other baselines ? (Especially SAM and UNets)\n- Why not mention nnUNets, and models specialized for tubular structures ?\n- Were the augmentations applied to all baselines as well? \n- Would it be possible to see standard deviations to assess the significance of the results?\n\nLIMITATIONS:\nThe authors mentioned the added complexity and computation time of their method in the supplemental section. I think it would be great to quantify this numerically both in terms of parameters and in terms of training time (for both phases).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a structure-aware sparse expert framework, named FineSAM++, which enhances foundation segmentation models. The proposed method has a gating module that identifies structurally uncertain regions and activates a Residual Expert to correct local structural inconsistencies of vessel segmentation. The proposed method has been demonstrated on five vessel segmentation datasets, and the results show superior performance compared to the existing state-of-the-art segmentation methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed framework with a gating module and residual expert refines the vessel segmentation mask from SAM model effectively.\n- The proposed method trains a part of the neural networks, which enables efficient training of the large model.\n- Experimental results using five public datasets demonstrate the effectiveness of the proposed method in local correction of the segmentation mask.\n\n### Weaknesses\n- There is no comparison for the size of learnable parameters between the proposed method and the existing SAM-based methods.\n- The proposed gating module requires a pre-defined error threshold (\\delta), which may make the model sensitive to the threshold value.\n- There is no study on the hyperparameters used in the gating module and residual experts, such as normalized routing weights (I, w_j) and the threshold (\\delta).\n\nQUESTIONS:\nPlease refer to the above weaknesses.\n\n- Is the proposed method efficient in terms of the number of learnable parameters among the SAM-based segmentation methods?\n- How is the error threshold (\\delta) in the gating module defined? Is the threshold the same for all five public datasets?\n- The hyperparameter study would strengthen the contribution of the proposed method.\n\nLIMITATIONS:\nyes (in supplementary material)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}