Text-promptable Propagation for Referring Medical Image Sequence Segmentation

Authors: Runtian Yuan, Jilan Xu, Mohan Chen, Qingqiu Li, Yuejie Zhang, Rui Feng, Tao Zhang, Shang Gao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Referring medical image sequence segmentation, Text-promptable propagation

Abstract:
Medical image sequences, generated by both 2D video-based examinations and 3D imaging techniques, consist of sequential frames or slices that capture the same anatomical entities (e.g., organs or lesions) from multiple perspectives. Existing segmentation studies typically process medical images using either 2D or 3D methods in isolation, often overlooking the inherent consistencies among these
images. Additionally, interactive segmentation, while highly beneficial in clinical scenarios, faces the challenge of integrating text prompts effectively across multimodalities. To address these issues, we introduce an innovative task, Referring Medical Image Sequence Segmentation for the first time, which aims to segment the referred anatomical entities corresponding to medical text prompts. We
develop a strong baseline model, Text-Promptable Propagation (TPP), designed to exploit the intrinsic relationships among sequential images and their associated textual descriptions. TPP supports the segmentation of arbitrary objects of interest based on cross-modal prompt fusion. Carefully designed medical prompts are fused and employed as queries to guide image sequence segmentation through
triple-propagation. We curate a large and comprehensive benchmark covering 4 modalities and 20 different organs and lesions. Experimental results consistently demonstrate the superior performance of our approach compared to previous methods across these datasets. Code and data are available at https://anonymous.4open.science/r/TPP/.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729958991333

thread_type:
official_review

<thread_object>
creator: reviewer_ilhu

<content>
summary:
This paper introduces medical image sequence segmentation, alongside Text-Promptable Propagation (TPP), which segments anatomical structures in sequential medical images guided by text prompts. TPP integrates cross-modal prompt fusion and a transformer-based triple propagation strategy. The method is developed for 2D and 3D medical image sequences with text-based references. For the dataset, the author curate a testbed with different imaging modalities and anatomical entities. Experimental results show that the proposed TPP is promising.

strengths:
Originality
The paper introduces a novel approach in both application and methodology. The originality lies in addressing Referring Medical Image Sequence Segmentation, which combines medical imaging sequences with medical text prompts to perform segmentation. The proposed TPP)segmentation model leverages text-based guidance for segmentation across both 2D and 3D medical image sequences, creating a cross-modal approach that integrates visual and linguistic information. 

Quality
The quality of the work is reinforced by its comprehensive, well-curated dataset that spans 18 diverse medical datasets across 4 modalities (MRI, CT, ultrasound, and endoscopy). The scope includes 20 different organs and lesions, encompassing a wide range of anatomical and pathological structures, which bolsters the reliability and generalizability of the proposed model. 

Clarity
The paper is well-written, with a logical structure that guides the reader through the problem setup, methodology, and results. The clarity is further enhanced by comprehensive illustrations of the model architecture and segmentation results, allowing readers to follow along without extensive prior knowledge. 

Significance
It explores the task of medical image sequence segmentation with text-guided prompts, addressing an essential need for context-aware segmentation in clinical settings where target structures may vary widely.
The development of the TPP segmentation model demonstrates a robust, adaptable framework that can interpret and propagate segmentation instructions across sequential medical images.
It curates a large-scale, diverse dataset specifically designed for this new task, contributing a valuable resource that could drive further research and improvements in text-promptable segmentation models for healthcare applications.

weaknesses:
The results lack fairness due to the absence of comparisons with stronger SOTA baselines, such as but not limited to UNeXt, 3D UX-Net, SwinUnet, and UNetR. Current baselines in the paper are comparatively weak, which does not fully substantiate the advantage of the proposed TPP model. Specifically, for the BTCV dataset, SOTA methods have reported Dice scores close to 0.8, whereas the proposed method significantly underperforms. Adding a comparison with these robust SOTA models would provide a clearer picture of the TPP model’s effectiveness. 

The proposed method may be overly complex for its purpose. As mentioned, this intricate design might not necessarily outperform simpler architectures that operate in fully or semi-supervised settings without relying on text prompts.

The paper does not clearly explain the advantages of treating 3D volumes as sequential data rather than applying a direct 3D model. A direct 3D model would likely capture context and spatial relationships more explicitly, whereas the sequential approach might weaken the model’s ability to leverage 3D spatial coherence. Not it lacks of theoretical or empirical comparison to justify the choice of sequential processing.

The value added by text prompts is unclear. The current implementation uses pseudo-text sequences, which may not provide personalized or contextually enriched guidance for segmentation. This detracts from the potential significance of using text prompts. On the other hand, if the text is from the actual clinical reports and from the same person with the image, it would make the model understand the personalized differences and add clinical value.

It is unclear whether the authors trained separate models for each of the 18 datasets or a single model with all datasets.

questions:
Why not stronger baselines?

Is this method overcomplicated for medical image segmentation?

Why it is better than 3D models, even without text?

Why the general description of the organ would provide significant information for segmentation, without seeing personal radiological report?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730104124861

thread_type:
official_review

<thread_object>
creator: reviewer_bfv2

<content>
summary:
This paper introduces a novel task, "Referring Medical Image Sequence Segmentation," aimed at segmenting specific anatomical structures or lesions in medical image sequences based on text prompts. To tackle this task, the authors propose a robust baseline model, Text-Promptable Propagation (TPP), which leverages temporal and cross-modal relationships within image sequences to achieve precise segmentation guided by text prompts. The key contributions include a cross-modal prompt fusion technique that integrates text and image information and a Transformer-based triple-propagation strategy that utilizes spatial and temporal consistency for accurate object tracking across sequences. Additionally, the authors curated a comprehensive benchmark dataset, Ref-MISS, covering diverse imaging modalities and anatomical entities, and demonstrated the superior performance of the TPP model through extensive experiments.

strengths:
1. The paper is well-structured and presents a novel approach to medical image sequence segmentation, making it overall clear and logically organized.
2. The method presented in this paper is quite innovative, particularly in its use of cross-modal prompt fusion and triple-propagation techniques for referring medical image sequence segmentation.
3. The Medical image sequence datasets covers 4 modalities and 20 anatomical entities, which is a large and relatively comprehensive.

weaknesses:
1.  A notable concern is the assumption that 3D imaging slices and video frames can be processed uniformly. While this may be technically feasible, it raises questions about practical applicability since 3D slices are typically evenly spaced, whereas video frames are often sampled or held at irregular intervals. This discrepancy might impact real-world usability, especially when temporal consistency is essential. A deeper analysis or justification of this choice, addressing its implications for varied temporal resolutions, would strengthen the method’s practical relevance.
2. The method adds the use of text prompts for segmentation, but it's worth questioning the added value of prompts given the effectiveness of fully supervised segmentation methods in medical imaging. Current fully supervised models achieve high accuracy without the need for additional prompts, especially in standardized medical datasets. The practical advantage of integrating prompts is not fully addressed, and further justification is needed to clarify whether prompts enhance segmentation accuracy, adaptability, or clinical interpretability in meaningful ways beyond what traditional supervised models provide.
3. The use of three prediction heads (box, mask, and class) in the proposed model is an interesting design choice, but the technical rationale for including both box and mask heads could be further clarified. Since the mask head inherently provides pixel-level precision, it seems redundant to have a box head, as the bounding box is generally a less precise representation. A deeper explanation of the box head’s role, particularly regarding how it contributes to the model's performance or stability during training, would be valuable. For example, if the box head aids in providing a coarse location as an initial reference for mask prediction, or if it enhances the model's ability to generalize across various object sizes, this should be explained to justify its inclusion alongside the mask head.
4. The selection of comparison methods in the experiments lacks representation of the latest state-of-the-art models. Notably, there is no comparison with recent benchmark methods like nnU-Net, which is widely recognized for its performance in medical image segmentation. Including nnU-Net or other recent high-performing models as baselines would provide a more robust evaluation and better demonstrate the advantages of the proposed method over current state-of-the-art techniques. This would enhance the credibility of the performance claims and place the proposed model’s effectiveness in a more competitive context.
5. The prompt experiments are a crucial aspect of this study, as they demonstrate the effectiveness and added value of incorporating prompts. However, the current experimental setup for prompt evaluation is relatively simple. Expanding these experiments would be beneficial, perhaps by examining various prompt types, specificity levels, or prompt designs to assess their impact on segmentation accuracy and adaptability. Additionally, testing on different anatomical structures or datasets could provide insight into how prompts contribute under varied conditions. This expanded exploration would strengthen the argument for using prompts and provide a clearer understanding of their practical advantages.
6. Some areas of expression contain minor ambiguities that could benefit from clarification. For example, terminology like "the referred object" (P5, L162-166) may not be immediately clear to readers, and consistency in using terms such as "Referring Medical Image Sequence Segmentation" would improve readability.

questions:
I'm a bit puzzled about the clinical significance of the new task proposed in this paper, 'Referring Medical Image Sequence Segmentation.' When performing a referring task, a physician typically already has preliminary information on the presence of certain lesions or diseases within the image sequence. This scenario seems somewhat inconsistent with the actual workflow of radiologists when conducting diagnostic assessments. Therefore, what is the medical relevance of the proposed task, 'Referring Medical Image Sequence Segmentation,' and in what specific scenarios could it be practically applied?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672853118

thread_type:
official_review

<thread_object>
creator: reviewer_sfow

<content>
summary:
The authors propose a new task: Referring Medical Image Sequence Segmentation, which aims to segment anatomical regions corresponding to given text prompts. To address this task, the authors propose a Text-Promptable Propagation (TPP) model, which takes sequential slices (either from 3D volume or videos) and text prompts as inputs and outputs the predicted masks. The authors have created a new dataset that consists of 18 3D/video medical datasets. Experiments were conducted against several referring video object segmentation algorithms, where the proposed method achieved better performance.

strengths:
1. The paper is well-written and easy to follow.
2. The experiments are thorough and demonstrate clear advantage over other referring video object segmentation algorithms. 
3. The proposed Triple Propagation that enforces temporal relationship between consecutive slices is novel.

weaknesses:
1. The motivation for Referring Medical Image Sequence Segmentation remains a bit unclear to me. Specifically, (a) although the authors propose a unified framework for 2D and 3D segmentation task and claim this to be an advantage this setting, no evaluation is conducted on 2D datasets; (b) although the closed set label space is a limitation of traditional segmentation, I don't think this will be a severe problem if the closed set covers most important region of an input (for example, [1] includes 25 organs and 6 types of tumors that cover all common organs and tumor types, making the closed set almost the full set of the label space). 
2. There is no ablation on the internal design of the TPP network, for example (a) how effective is the "Cross-modal Prompt Fusion" against a simple fusion strategy that averages or concatenates the image and textual features; (b) the prompt is repeated N_q times, meaning each image will have N_q outputs, and thus the computation cost is multiplied by N_q times as well. The analysis on the computation cost (such as FLOPS) is not included and the effect of N_q is not studied. 

[1] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection. https://arxiv.org/pdf/2301.00785.

questions:
My main concern is 1b. To address this concern, I'd like to see some comparison against 3D segmentation algorithms, such as [1] and [2]. The experiments can be conducted on a specific type of body locations (for example only training and evaluating on abdomen data) due to the limited time for rebuttal. 


[1] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection. https://arxiv.org/pdf/2301.00785. 
[2] nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. https://www.nature.com/articles/s41592-020-01008-z
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689855784

thread_type:
official_review

<thread_object>
creator: reviewer_qiah

<content>
summary:
This paper aims to solve the challenges of limited interaction between 2D and 3D segmentation models, in parallel of adding interactive prompt to provide human-guided context for segmentation in real clinical scenarios. This paper contribution can be summarized as follows:
1) Proposed an innovative task: referring medical image sequence segmentation
2) Proposed a baseline mode Text-Promptable Propagation (TPP) to exploit the intrinsic relationship among sequenctial images and thir associated textual descriptions
3) Benchmarked the model across 18 different datasets across 4 modalities.

strengths:
The strength of this paper can be summarized as follows:
1) Performed experiments with 18 datasets
2) Compared baselines with SAM-2
3) Aimed to create new task for medical imaging domain

weaknesses:
The weakness of this paper can be summarized as follows:
1) Really confused about the clinical scenarios or the medical problem that this paper is targeting
2) Limited experiments have been performance and haven't compared to the medical domain state-of-the-art 
3) Insufficient clarity on the innovation of the proposed model, it seems like this model is a composition of so many current design blocks

questions:
1) I am confused about the clinical problem that the paper are trying to solve, as 2D snapshots (e.g. CT) is possible to be demonstrated in the temporal domain (i.e. same subject but takes the image in different time), due to the needs of quick imaging in the clinical scenario. I am wondering if this is the problem that this paper are trying to solve?

As the medical image sequences you are referring in the paper is really similar to the consecutive slices of the same 3D image or video, it will be great to have more clarity on this.

2) Previous similar work have been demonstrated to adapt text-prompt across 9 modalities:
- Zhao, Theodore, et al. "BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once." arXiv preprint arXiv:2405.12971 (2024).

You can claim that your work is an extension idea from this, but I haven't seen any citation / experiments comparison with this. It will be great if you can add / use similar text scenario in BiomedParse to have a comparison.

3) In Table 4, as the experiment is performed with SAM-2, it should compare with the current SAM-based state-of-the-art model and even 2D completely supervised model (i.e. nnUNet, Swin-UNet), as the final goal is to enhance the segmentation performance for all slices input. 
- Ma, Jun, et al. "Segment anything in medical images." Nature Communications 15.1 (2024): 654.
- Isensee, Fabian, et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation." Nature methods 18.2 (2021): 203-211.
- Cao, Hu, et al. "Swin-unet: Unet-like pure transformer for medical image segmentation." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

4) As one the innovation on your model is to adapt descriptive text for enhancing the slice-by-slice relationship for segmentation, I also want to know the effectiveness of different versions of description, seems like there is no experiments to benchmark different versions of description, although you have provided the text prompt in the appendix. Wondering if this will be one of the core to affect segmentation performance.

5) Also, is your model generated binary segmentation and use text to refer the class semantics? Seems like you have used focal loss during training and I assume that the class label for the anatomy have been used. It will be great to see the performance if we don't need the class label for training and just adapt the text as a loss function.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new task, Referring Medical Image Sequence Segmentation, which aims to segment anatomical regions in medical image sequences based on text prompts. To address this, the authors propose the Text-Promptable Propagation (TPP) model, leveraging cross-modal prompt fusion and a Transformer-based triple-propagation strategy to exploit spatial, temporal, and textual relationships for segmentation. The task and method are claimed to address challenges in integrating 2D and 3D segmentation models and enabling human-guided context in clinical scenarios. A comprehensive benchmark dataset, Ref-MISS, comprising 18 datasets across diverse imaging modalities, was curated to evaluate the method. Experiments demonstrated that TPP outperforms existing referring video object segmentation algorithms.

Strength: The paper's strengths lie in the new approach for Referring Medical Image Sequence Segmentation, using cross-modal prompt fusion and triple-propagation techniques to address the need for context-aware segmentation. Besides, it introduces a large-scale, diverse dataset covering 18 public datasets across 4 imaging modalities and 20 anatomical entities, providing a valuable resource for future research. The focus on text-guided prompts for medical image segmentation highlights its practical relevance for varying clinical scenarios.

Weakness: Most reviewers raise the concern that the experimental evaluation is limited, lacking comparisons with state-of-the-art methods in the medical domain and important baselines. The assumption that 3D imaging slices and video frames can be processed uniformly may be questionable: the paper does not adequately justify why it is beneficial to treat 3D volumes as sequential data instead of using direct 3D models, which may better capture spatial coherence. Moreover, the experiments do not sufficiently validate the claimed contributions, particularly in 3D scenarios. The added value of text prompts for segmentation is unclear, given the effectiveness of fully supervised methods. 

Overall, considering the paper's contribution and the remaining concerns about the experiment evaluation and results comparison with regard to the claims, I suggest rejection and that the paper could be improved by a major revision.