CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation
Authors: Zhongzhen Huang
Yankai Jiang
Rongzhao Zhang
Shaoting Zhang
Xiaofan Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Promptable model
Visual-Textual prompt
Multi-organ and tumor segmentation
Primary area: machine_learning_for_healthcare
TL;DR: We intro a novel automatic model that coordinates anatomical-textual prompts for multi-organ and tumor segmentation.

Abstract: Existing promptable segmentation methods in the medical imaging field primarily consider either textual or visual prompts to segment relevant objects, yet they often fall short when addressing anomalies in medical images, like tumors, which may vary greatly in shape, size, and appearance. Recognizing the complexity of medical scenarios and the limitations of textual or visual prompts, we propose a novel dual-prompt schema that leverages the complementary strengths of visual and textual prompts for segmenting various organs and tumors. Specifically, we introduce $\textbf{\textit{CAT}}$, an innovative model that $\textbf{C}$oordinates $\textbf{A}$natomical prompts derived from 3D cropped images with $\textbf{T}$extual prompts enriched by medical domain knowledge. The model architecture adopts a general query-based design, where prompt queries facilitate segmentation queries for mask prediction. To synergize two types of prompts within a unified framework, we implement a ShareRefiner, which refines both segmentation and prompt queries while disentangling the two types of prompts. Trained on a consortium of 10 public CT datasets, $\textbf{\textit{CAT}}$ demonstrates superior performance in multiple segmentation tasks. Further validation on a specialized in-house dataset reveals the remarkable capacity of segmenting tumors across multiple cancer stages. This approach confirms that coordinating multimodal prompts is a promising avenue for addressing complex scenarios in the medical domain.

CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation
Zhongzhen Huang! 1.2, Yankai Jiang", Rongzhao Zhang?,
Shaoting Zhang' ( Xiaofan Zhang 12gg
'Qing Yuan Research Institute, Shanghai Jiao Tong University SShangaai Al Laboratory
(huangzhongzhen, xiaofan. zhang)asjju.. (aang))estu.edu.coo edu.
{jiangyankai, zhangrongzhao, zhangshaotinggbbpplb.ooggg cn
Abstract
Existing promptable segmentation methods in the medical imaging field primarily consider either textual or visual prompts to segment relevant objects, yet they often fall short when addressing anomalies in medical images, like tumors, which may vary greatly in shape, size, and appearance. Recognizing the complexity of medical scenarios and the limitations of textual visual prompts, propose novel dual- prompt schema that leverages the complementary strengths of visual and textual prompts for segmenting various organs and tumors. Specifically, introduce CAT, an innovative model that Coordinates Anatomical prompts derived from 3D cropped images with Textual prompts enriched by medical domain knowledge. The model architecture adopts general query-based design, where prompt queries facilitate segmentation queries for mask prediction. To synergize two types of prompts within a unified framework, we implement a ShareRefiner, which refines both segmentation and prompt queries while disentangling the two types of prompts. Trained on a consortium of 10 public CT datasets, CAT demonstrates superior performance in multiple segmentation tasks. Further validation on specialized house dataset reveals the remarkable capacity of segmenting tumors across multiple cancer stages. This approach confirms that coordinating multimodal prompts a promising avenue for addressing complex scenarios in the medical domain. Codes are available at https:://// /tttps://giite gig.ttt.bbb..... com cwtccom///ooo/ee comm/zongzi3zz/CAT
Introduction
Advanced prompt engineering (22 has augmented large language models 5 with emerging capabilities. However, these paradigms have not yet been successfully applied to vision tasks, primarily due the ever-changing and unpredictable natures of computer vision 66 7 This is particularly evident in medical domains, where variations in imaging protocols, noise/artifacts. and patient-speeifo pathologies pose significant challenges 9 10]. To tackle these challenges. there have been some efforts within the community to develop promptable models for segmenting objects in medical images: One type of such effort focuses on textual- prommpee models [2 [3], which show profound competencies in segmenting specific organ or tumor referenced by arbitrary text phrases. These approaches involve distilling knowledge from language models like CLIP 14 or
BERT [15 to facilitate the alignment between visual and textual representations; 2) Another direction in promptable segmentation research aims to visual-prompted models [66 [1 18 19 20 21, which
rely on visual examples visual landmarks (e.g., boxes and points). Recently, the trend toward visual-prompted segmentation models in the medical domain mainly focuses on fine-tuning SAM 22 with lightweight, plug-and-play adapters or modifying SAM into 3D-based architecture
*Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

T-Stage III
T-Stage IV
Number of Caves
Lour Tomer
Colos Tunnrr
Tiver
The T-stage primanily indicates exient to which invades its nearby tissues. more advanced stage suggests deeper invasion and patentially greater spread within the local
Colon tumoes can present varying densities might be associated with surrounding in inflammaaiin,, adjacent organ invasion, or regional lymph node colargement.
Organ
Anomaly
Category
Colon Tumor in Different Cancer Stages
Figure 1: Left: Long-tailed curve of the category and the number of available cases that can be obtained in the medical field. Right: Tumors in different cancer staging with diverse shapes and sizes.
Despite the promising advancements brought by the prompt-enhanced segmentation paradigm, those segmentation models still face significant limitations. As illustrated in the left part of Figure T the distribution of medical datasets often follows a long-tail pattern, where an increase in the diversity of detectable anomalies corresponds with sharp decline in the number of available cases. Textual- prompted methods utilize textual representations from referred text phrases to guide the segmentation process, requiring alignment between visual and textual representations, Although descriptive texts can cover intricate and rare anomalies with domain knowledge, data scarcity due to long-tailed distribution hinders the effective learning of alignments between textual and visual representations. This issue is particularly significant in the medical domain, where numerous corner cases (e.g., tumors with variations in shape, size, density distribution, and blurring boundaries) need to be addressed. Conversely, visual prompts are not constrained to the need for cross-modal alignment, providing a more intuitive and direct method enhance the segmentation process. Nonetheless, visual prompts fail to convey the general concept of cach object, leading to performance drop when confronted with various scenarios medical domains, especially for tumors. For instance, colon tumors range from different cancer stages [23] and have significant inter- and intra-patient in tumor sizes and shapes, as shown in the right part of Figure Moreover, tumors the same stage can exhibit varying densities. Given the inherent diversity in tumors, there is necessity to provide comprehensive knowledge of each tumor type via textual descriptions.
In this work, we strive to develop a promptable segmentation model that utilizes the strengths of both visual and textual prompts without human interaction, aiming a fully automatic model for medical professionals. We propose a new dual-perspective prompting scheme. On the one hand, we directly employ the cropped volumes derived from the anatomical structure as our visual prompts. We refer to such prompts as anatomical prompts, intending to represent target objects in a more intuitive and visually coherent manner. On the other hand, we enhance the textual prompts with more comprehensive knowledge. With the proposed prompting scheme, we introduce CAT, a model towards comprehensive segmentation that harnesses a dual- poom-pioottggggg mechanism to Coordinate Aatomical and Textual prompts.
CAT follows the general query-based design with two extra parallel encoders dedicated to processing anatomical and textual prompts into prompt queries. Inspired by previous segmentation models [24]
adopt query embeddings perform mask predictions. A ShareRefiner is utilized to refine segmentation queries and prompt queries by attending them to image features from the backbone. We employ two distinct feature assignment strategies for prompt queries, ensuring that these prompt queries are disentangled for more versatile representations. Following the refinement process, PrompiRefer updates segmentation queries by integrating both types of prompt queries. The mask prediction module then generates binary masks by performing a simple dot- not-product of segmentation query embeddings with the pixel embedding map obtained from the backbone. To further facilitate the coordination, we augment CAT with cross-modal alignment between anatomical and textual prompts. By training on an assembly of 10 public CT datasets in abdomen, CAT demonstrates strong segmentation capabilities on these datasets and achieves remarkable results on an in-house dataset encompassing four cancer stages. Further ablation analysis shows that anatomical and textual prompts serve complementary roles. To summarize, contributions are threefold:
25 26 27..

---

We design new prompting scheme that utilizes both complementary strengths of anatomical prompts and textual prompts for medical image segmentation.
We build CAT, fully automatic and promptable model that coordinates anatomical prompts derived from 3D cropped volumes with textual prompts enriched by medical domain knowl- edge, enabling strong flexibility for various segmentation tasks.
Extensive experiments demonstrate the benefits of coordinating anatomical prompts and textual within one model. CAT consistently achieves state-of-the-art performance on multiple segmentation tasks and has generalization capability to diverse tumor types.
2 Related Work
2.1 Textual-prompted Segmentation in Medical Imaging
Prompt engincering has T2 achieved remarkable progress in natural language processing and shown great potential in general visual perception 28
7 31. By leveraging pre-trained vision- language foundation models [14. lextual-promptr methods demonstrate impressive capabilities in 34
29
30
24
36 These advancements in the natural image field have significantly propelled the field of 2D medical image segmentation [37], where the innovative model architecture is used to recognize a variety of visual contexts through textual prompts. Recently, some efforts tried to train textual-pron promppee universal model for segmenting various organs and A pre- tre-trained text encoder is adopted to encode the injected prompts and guide the target grounding process. Moreover, the latest attempts [12] focused on extending the short textual phrases with knowledge. However, the reliance on texts struggles with medical image segmentation due to the potential misalignment between textual descriptions and complex visual patterns. To solve the limitations caused by linguistic ambiguity, we integrate visual inputs (i.e., anatomical prompts) for more accurate and comprehensive image perception.
open-vocabulary segmentation 31 33
35
tumors in 3D volumes [1 12
2.2 "isual-prompted Segmentation in Medical Imaging
Beyond textual- prompted methods, the field has seen a notable shift towards incorporating visual prompts enhance accuracy and context sensitivity [16] Different from textual prompts, visual prompts introduce more flexibility and context-awarenes. for models. The innovative Segment Anything 22 introduces a promptable model for generalized image segmentation, Building upon the foundation model, the field has seen substantial shift towards applying this paradigm to the medical domain. One line adapts SAM to general medical image segmentation with fine- tuning [17 19 20 18). Some works like SAM-Med2D [19 and 3DSAM-adapter [20] utilize adapters (38 to transfer the capabilities of SAM to medical images with a few number of trainable parameters. Other line (39] trains 3D models from scratch straightforwardly. SAM-Med3D 40)
and Vel (39] developed 3D SAM models from scratch by reforming the 2D model and training with large scale of CT scans. Incorporated with higher annotated ratio, 3D prottprropttble model CT-SAM3D 41 is introduced with a progressively and spatially aligned prompt encoding technique. Nonetheless, due the substantial disparities among anomalies in the medical domain, visual prompts could not provide a generic concept for each type. Our work resembles the textual and anatomical prompts support the multi-organ and tumor segmentation.
3 Method
In this paper, we focus on applying anatomical prompts (i.e., cropped 3D medical volumes) and textual prompts for generic segmentation tasks in the abdomen involving organs and tumors. CAT employs popular query-based decoder architecture with a sophisticated interaction paradigm between queries and prompts, aiming at predicting N categories in the abdomen, as shown in Figure Our model integrates four main components: i) Vision Backbone, designed to extract image features and construct pixel embedding maps, ii) Prompt Encoders, used to encode anatomical and textual prompts provided by users, respectively, iii) ShareRefiner, utilized to refine segmentation queries and prompt queries, and iv) PromptRefer, generating target queries for prediction.
3

---

PromptRefer
Allgweeer
ShareRefiner
Colon
Colon Tumer invading Coion Inzestine (b)
...
Qr
Anatomicall Prompt Encoder
Textual Prompt Encoder
Vision Backbone
Time
highh repol dion Fase map
Aetsone have neteler skogee
Colos sulddrr shape
22 Crrpped PaAaNN
Tomes
AANNY organ
(0]
PromytRRlle
Figure 2: (a) CAT' follows the query-based segmentation architecture. 3D cropped volumes according to the anatomical structure are utilized as anatomical prompts. Texts enhanced by professional knowledge are adopted as textual prompts. Learnable queries and both prompts are utilized for the final prediction via SeareRefiner and PromptRefer. (b) The case of colon tumor in Stage-IV invading the intestine. (c) Attention masks in PromptRefer for assigning specific prompts queries.
3.1 Vision Backbone and Prompt Encoders
Backbone. Following other unified models 226
42
for segmentation tasks, we perform mask
classification for all organs and tumors. We adopt key idea from Mask2Former [42] [1] to construct pixel embedding map, which is obtained from the backbone encoder features. As shown in Figure 2 the vision backbone comprises vision encoder Enc responsible for extracting multi-scale visual features V and vision decoder represented as Dec, which gradually upsamples visual features to the high-resolution pixel embedding map O. Given an input 3D volume E H WxD the process can be formally represented follows:
V Enc(I), Dec(V);
where V [Vile Vi HT xw w D'x and the number of layers of the vision encoder. Here, H' W D' and C denote the height, width, depth and channel dimension of V respectively. O E HxxxWDxxCC is obtained by fusing feature maps from the Enc, C, is the dimension of O.
Prompt Encoders. Prompts have been widely used in segmentation tasks recently. Textual prompts provide comprehensive concepts for the target object but bring some ambiguities, such the fact that both the intestines and the colon have "tubular shape". Conversely, visual prompts are more helpful disambiguating the user's intent when textual prompts fail to identify the correct object. However, these visual prompts have limited robustness in handling large variations in medical images [41]. Our method enhances the segmentation process via the coordination of anatomical and textual prompts. Specifically, the anatomical prompt is referred region from another CT exam, which can be particular tumor or cropped volume of the same semantic concept. Given N cropped volumes PAj E IR Haxxaxx WaxDA, .j E 21,2.... 1,.,,...N,, N}, we encode these volumes through a pre- pre-trained encoder Enc and obtain anatomical prompt embeddings E.. To convey comprehensive notion of targets with detailed information, we utilize long textual descriptions previous work 112, we incorporate textual prompts with medical domain knowledge. employ text encoder Encr 43] to encode these descriptions and use the [CLS] token output the textual prompt embedding, denoted as Er. The process can be denoted as follows:
our textual prompts. Following
E = Enc (P A), Ey = Ency (Pr) Q Linear EA), Q Linear (Er),
(2)
(3)
where QA, QT NxC are prompt queries and C' the channel dimension.
3.2 ShareRefiner
CAT predicts N masks based on the learnable segmentation queries Qss and the pixel embedding map O. Our initial steps involve feeding both segmentation queries Qs and prompts queries QA. Qr

---

into the proposed ShareRefiner to enable the queries to interact with multi-scale features. In our approach, queries will selectively attend local image features for self-refimmmens. Specifically, the ShareRefiner consists of a series of cross-attention blocks where queries perform cross-attention with respect to the target visual feature V To disentangle these queries, we adopt soft and hard assignments for [Qs, Qr) and QA respectively. A hard cross-attention layer is applied to extract anatomical prompt features from the multi-scale feature maps, ensuring that each anatomical query gathers discriminative visual regions without overlaps. For the i-th layer of ShareRefiner, the hard assignment similarity matrix Q' is computed via Gumbel-Softmae
as follows:
S = Q's Flatten(V"))
(4)
St gumbel Softmax ((5t + G*) /T)
(5)
St onchot
Onehot (argmaxN (S (Seumbe))
(6)
Where St G E NNNH''I G' are i.dd random samples drawn from the GuumbelOO, 1) distribution and is learnable coefficient. The one-hot operation of the argmax is performed over Sgumbel for hard assigning. Since the straightforward hard assignment (i.c., one-hot) is not differentiable, the straight-through trick in 46 [47 is used compute the assignment similarities S of one-hot value:
S (Smeeaa)
St gumbel sg (S Sumbelt)
where sg denotes the stop gradient operator. Following the feature grouping process, we use self-attention layer to regulate the relationships among queries and a feed-forward layer for projection. Formally, the aforementioned process is represented as:
Q' s. Q' r ShareRefiner Qs: Qr), V, hard = False),
(8)
Q' A = ShareRefiner QA, V, hard = True).
(9)
3.3 PromptRefer
In practice, when segmenting target objects, more attention needs to be paid to the relevant context. Localizing the typical tumor requires being aware of the anomalous features the relevant organ, and even identifying organs requires focusing on the anatomical structures involved. For instance, as shown in Figure colon tumors of Stage-IV would invade the adjacent organs like intestines. Di- rectly combining prompt queries for mask prediction is suboptimal since there is still inherent gap between the two types of prompt embeddings even though we have unified them into prompt queries. The typical objective is to classify Q's into respective regions with the guidance of prompts. To this end. we introduce PromptRefer implemented by the cross-attention mechanism with carefully crafted attention masks in Figure Cc. Here, group of prompt queries {Q' Ai:..., Q'T): D'T)..... ... is employed to specific segmentation query. To ensure the distinction of queries, segmentation queries required to only interact with queries in the group. This is illustrated in the following equation:
s PromptRefer(() s Q' A. Q' rl. mask).
(10)
Here, Og represents the decoded segmentation query features for predicting masks.
To further integrate two types of prompts and push segmentation queries closely the referenced prompt, we employ query-level contrastive learning in our model. Specifically, given the decoded segmentation query features and refined prompt queries, we define two types of losses and calculate the InfoNCE loss
[14) as follows:
1
N N explŌs: E' ('(AATT)) Lppp = Σ
exp( N Q'T.)
C.2p
(11)
1=1 = E exp(Os: Q' (AAAT))))
N
1=1 E j= exp( Ai Q'T,)
Where Ös, Q'A. Q'T are derived via linear projection layers. The contrastive alignment can be regarded distillation process, whereby each prompt query contributes to the segmentation queries. It also benefits the knowledge exchange between queries from two modalities.
3.4 Training Objective and Strategy
Training Objective. The binary mask proposals M E [0, NxXHXXX are calculated through the multiplication operation between the decoded segmentation query features Os and the high- resolution pixel embedding map O followed by Sigmoid. We employ the dice loss for mask
5

---

prediction. For classification loss, following [3], we adopt the cross-entropy loss that measures the difference between predicted objects and the categories. Moreover, the contrastive loss is applied to the similarities among queries. The final loss takes the following form:
Liotal
Ldice + Cets + Lspp + Lypp.
(12)
Anatomical prompt training strategy. For anatomical prompts, we leverage the bounding box derived from masks in the public dataset and relevant anatomical structures to crop set of prompt volumes for each category and unify these volumes into the same size. To alleviate the domain shift, we use a pre-trained encoder 48 for feature extraction. During training, we randomly sample an instance from the set of prompt volumes for each category, excluding the identical one.
Textual prompt training strategy. For the textual prompt, we identify the category names in the abdomen. We call GPT-4 [49] to generate descriptions with medical domain knowledge for each category more than 20 times. We recruit one board-certified physician rewrite the description according to the results. Besides long descriptions for each category, we also construct several short textual templates. We use the long description for the positive category and randomly sample short phrases for those negative categories. More details can be seen in the Appendix A.T
4
Experiments
Dataset and Settings. CAT' is trained on the curated dataset from 10 public datasets 50 51 52 53
58 59), which contain multiple organs and tumors in the abdomen. Following settings we assemble this dataset with the data pre-processing to reduce the domain gap among various datasets. In the testing phase, two public datasets and one in-house dataset are used for evaluation. The ho-house test dataset contains 80 3D CT volumes with colon tumor masks, ranging from Stage-I to Stage-IV. For evaluating the accuracy of organ segmentation, we employ FLARE22 [60] as the external test set. For tumor segmentation, private dataset. Details of datasets are in Appendix A.2
54
55
56
57
in 13
utilize the combination of MSD dataset [59 and the
Our model follows the query- -based segmentation architecture 44 26). That is, we use Swin UNETR [6] as the backbone and Clinical-Bes [4] as the text-encoder. We follow the experiment settings of previous work [3 12]. Details are in Appendix AA.3] For the evaluation, the Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance
Implementation Details and Evaluation Metries.
(HD95) are utilized to gauge the performance of organ/tumor segmentation.
4.1 Main Results
Organ Segmentation. In this study, we explore the organ segmentation performance of our model on the external testing set of FLARE22. We present the detailed organ-wise segmentation results in Table [ Compared to models [7 19 adapting SAM [22) to the medical fields, our approach yields substantially better performance across all 12 organs. For instance, CAT significantly surpasses MedSAM [T7] by large margin of 21% DSC points on the adrenal gland segmentation and 15% DSC points on the esophagus segmentation. Such results show the gap in applying 2D SAM-based models for 3D volumes, especially for objects with small sizes or intricate shapes. When compared
to 3D SAM models 41 39 40). our model performs significantly better than models 39
40. which
take in visual prompts like SAM. For example, we surpass SAM-Med3D 40) and Seg Vol 39 by 30% points and 20% points in terms of the average score. We can observe that these models also struggle with the problem of handling some difficult organs. In addition, just with one prompt for each category, CAT achieves comparable results with CT-SAM3D [41, which utilizes spatially aligned prompts progressively in multi-rounds. Owing to the PromptRefer module, our model demonstrates superior performance in organs where tumors frequently occur. Specifically, CAT leads by 2% points in the liver and by 6% points in the pancreas. CAT also demonstrates superior performance compared with recent
models 13 12). Although the gap of organ segmentation is only 2% points, we observed that they do not generalize well on the tumor segmentation task without more intuitive visual prompts, as described in the following part. More details are shown in Table
Tumor Segmentation. We do the evaluation on MSD dataset (59 and an in-house dataset to validate its tumor segmentation capability. 2 shows the comparison results of four tumor categories. We do not report HD95 scores for SAM- based methods since this comparison may not be entirely

---

Table I: Organ segmentation performance FLARE22. The results(%) are evaluated by DSC.
Scores of SAM-based are adopted from the CT-SAM-Med3D [41]. f denotes obtained via the official pre-trained weights. means implemented from the official code and trained on the same dataset. Abbreviations: "Live"-Liver, R_.id."- Kidney, "Spl." "Pan." "aanc--a,,,,,s, "Aor." Aorta, "IfCrr-nferrir Vena Cava, "RAG" Adrenal Gland, "LAGG"-Lef Adrenal Gland, "Gal."-
Gallbladder, "Esoo.-FEsohhaa
"Sto." -Stomach, "Duo,"-Duodemin "-Duodnuum,
Kid."-IIfeee Vena Cava.
Methods
Liv.
R Kid.
Spl.
Pan.
Aor:
IVC
RAG
LAG
Gal.
Eso.
Sto.
Duo.
Kid.
Avg.
SAM
[2
86.0
87.6
84,5
53,4
77.5
44.5
19,4
33.9
52.4
35.2
68.0
44.4
82.6
59.2
MedSAM Iv
93.0
90.0
89.1
73.5
82.5
76.5
36.0
48.7
56.4
64.7
84.0
53.9
89.7
72.2
SAMDMed2D
19
91.4
83.7
83.9
58.8
60.6
18.6
10.6
27.1
32,9
28.1
72.9
45.4
86.0
53.8
5AM-Med3D
46
85.4
84.7
46,9
60.4
44.5
32.6
35.3
56.0
32.6
46.9
84.9
55.5
SegVol 39
83,9
71.7
75,9
69,4
83.1
80.3
42.1
49.7
55.6
69.6
81.1
55.6
75.1
68.7
CT-SAMBD
41
95.6
95.0
96.1
83.6
94,5
91.8
78.4
82.5
88.4
82.9
92.3
73.2
94.8
88.4
Universal
97.4
95.5
96.4
73.7
84,9
84.4
72.9
73.4
86.0
76.8
88.5
74.5
96.9
84.7
7cPT" 12
96.7
95.6
96.6
84.3
90.0
84.4
67.2
66.8
79.6
74.2
85.2
59.1
97.2
82.8
CAT
97.7
96.3
97.1
89.2
90.5
88.0
73.6
74.3
83.0
80.1
88.2
73.4
97.3
86.8
Table 2: Segmentation performance (%) of tumors on MSD [59] and In-house dataset. We compare our method with traditional and promptable methods. t denotes obtained via the official pre-trained weights. * means implemented from the official code and trained on the same dataset.
MSD Dutaset (Tumor in Abdomen)
In-house Data (Colon Tumor)
Method
Liver
Pancreas
Hopatic Vessel
Colon
T2
T3
Avg.
DSC1
HD9S]
DSC1
HD95]
DSC1
HD9S_
DSCT
HD95,
DSC
DSCT
HD995]
unUNNtt [2
66.42
42.29
43.50
25.80
66.90
47.59
41,41
153.06
19.51
45.06
44.87
15.54
43.00
1500/18
Swin UNETR"
48
42.54
41.77
22.87
63.32
39.35
161.26
21.40
33.32
46.11
52.72
45.92
168.25
SAM M
40
44.78
40.05
44,86
39.23
34.28
42,65
50.20
42.65
47.11
SogVol
66.20
16.36
68.57
60.63
36.93
42.63
60.17
49.53
50.28
Universalf
13
65.68
63.31
15.72
16.58
66.31
51.17
42.26
115.40
7.11
43.28
46.52
$3.08
47.14
1440.28
ZePT
12
68.58
43.23
44.39
19.47
68.12
33.94
40.38
113.07
23.87
34.64
50.81
$1.09
46.28
155.83
CAT
72.73
34.64
49.67
15.56
70.11
108.26
30.62
45.61
55.85
57.37
53.35
80.96
equitable. Our CAT outperforms baselines [62 48) significantly across all tumor subtypes, achieving at least a 4% improvement in DSC. For the comparison of SAM--ased methods [40] [39] it is worth noting that only one point derived from the ground truths is utilized as the prompt during the inference time since tumors of small sizes are difficult to identify but can be easily segmented with the provided prompts. Without using information from labels, CAT still achieves better performance except for the colon tumor segmentation, as evidenced by 5% improvement in the DSC of the pancreas tumor. These results validate the proposed PromptRefer module (i.e., attending to specific organs when segmenting tumors). Moreover, we compare CAT with textual-prompted methods. CAT surpasses the previously across four tasks by 4% in the average DSC. As our model is an extension of textual- prompted methods, these results demonstrate that incorporating the anatomical prompts can produce higher-quality masks for tumors. More details are shown Table
We further conduct the evaluation on an in-house colon tumor dataset, with tumors ranging from Stage-I (T1) to Stage-IV (T4) according to the Cancer Staging System (23 As shown in Table 2 CAT showcases strong capabilities in dealing with tumors of diverse sizes and shapes. To provide deeper insight, we present the average score of each subtype. It is noted that the higher the number after the "T"" the larger the tumor or the more extensively it has grown into nearby tissues. We can observe that SAM-derived methods perform well in segmenting relatively small T1 tumors with point prompts. However, Seg Vol (39. which achieves the best results in MSD colon tumor segmentation, struggles to handle tumors that invade nearby organs or tissues. is evident from the performance gap between T3 and T4 tumors. In these cases, tumors in each instance can present with varying densities and show distinct shapes at different locations. Only using visual prompts without comprehensive descriptions to cover all abnormal parts proves challenging. Such scenarios are vital in the medical field, highlighting the need for further research in applying visual prompts for tumor segmentation. In this paper, we coordinate anatomical and textual prompts along with the predefined attention mask to alleviate this problem. CAT outperforms other models by at least absolute 7% DSC in T4 and 3% DSC on average, demonstrating much better generalizability and robustness. This result suggests that leveraging anatomical prompts together with medical domain knowledge is an alternative way to address intricate scenarios in the medical domain.
7

---

Zoom-in Image
Ground Truth
Ours
Universal
SegVol
Zoom-In Image
Ground Truth
Ours
Universal
SegVol
Figure Qualitative visualizations of the proposed model and other prompting methods on or- gan/tumor segmentation. The segmentation results presented from rows one to five correspond, in order, to the duodenum, liver tumors, pancreas tumors, colon tumors, and colon tumors Stage-IV.
Qualitative Comparison. To give more intuitive comparisons, we present the qualitative results in Figure 3 In the first row, we observe that current methods may struggle with segmenting tubular- shaped organs, particularly SAM-derived methods like Seg Vol [39,, since visual prompts can not convey the general concept of such tubular-shaped objects. The results in the second and third rows reveal that textual-pronn proomped models are prone misclassification and overlook key details due the lack of intuitive context, underscoring the importance of visual prompts. However, these only relying on visual prompts struggle with tumors of intricate shapes without the support of medical domain knowledge, shown in the fourth row, where Seg Vol exhibits high number of false positives normal CT scans. When tumors invade other tissues, existing models often fail due to segmentation target incompleteness and misclassification of normal regions tumors. In contrast, CAT can precisely identify most abnormal regions and consistently generate results that are more consistent with the ground truth compared to all other models.
4.2 Ablation Studies
To fully investigate the contribution of our introduced components, we conducted ablation studies to compare each part in both organ and tumor segmentation tasks. We selected subset of organ and tumor categories that are particularly challenging for models, facilitating more intuitive comparison.
Ablation of Two Prompts. We start from basic query-based segmentation architecture without any extra prompts, and then we ablate the effectiveness using different prompts for different tasks. As demonstrated in Table (the first four rows), eliminating the prompt schema leads substantial performance drop on both organ and tumor segmentation (64.08 - 54.39 in Duodenum and 72.49 66.37 in Liver tumor). The observed declines in the third row indicate that textual features, which are rich in semantics but lack appearance features, may not generalize well to objects with intricate structures or small sizes. It is evidenced by performance reduction of 6% in Esophagus segmentation and 4% in Hepatic Vessel tumor segmentation. Moreover, we can observe from the second row that the tumor segmentation capability is inadequate when relying solely on anatomical prompts. This issue stems from the diversity and variance of tumors. For instance, every example the model encounters is drastically different tries to identify the location and nature of pancreas tumor. Although tumors can be regarded as anomalies, the lack of consistent context poses a challenge for the model in establishing general concept solely with the cropped volumes. The coordination of anatomical and textual prompts improves the performance of the promptable model. This confirms the efficacy of our joint prompts schema, which is designed help the model form more stable and generalizable masks in the medical field.

---

Table 3: Ablation studies of two prompts and model designs organ and tumor segmentation dataset.
Variant
Organ (%)
Tumor (%)
Hard
Mask
Pan.
RAG
LAG
Eso.
Liver
Pancreas
HepVes.
Colon
T4
78.18
69.63
69.08
76.99
54.39
66.37
42.05
62.20
39.85
51.17
83.55
72.80
71.65
79.31
60.45
64.82
45.08
68.72
43.84
53.91
80.62
71.02
70.34
72.81
57.31
69.13
44.31
65.18
40.16
52.32
83,50
72.71
69.96
78.99
64,08
72.49
44.55
69.40
44,50
55.84
86.74
72.41
69.00
77.68
59.99
69.12
43.23
67.75
41.32
54.33
87.36
74.46
74.02
75.39
70.80
72.64
48.49
69.02
47.29
53.67
88.49
73.24
74.51
80.76
70.26
72.18
46.46
69.97
46.65
58.49
88.
74.42
72,50
79.30
71.26
70.95
45.52
69.51
46.07
56.41
89.24
73.69
74.63
80.10
73.46
72.73
49.67
70.11
48.31
57.37
2
Segmenatico Query Paaurrn
Teracl Pronpe
Figure 4: T-SNE visualization of the distribution of Features. Left: Two types of prompt embedding before and after refinement. Right: Segmentation query features with and without constrastive alignment. (1-9: right kidney, left kidney, liver, pancreas, colon, kidney tumor, liver tumor, pancreas tumor, colon tumor).
Effectiveness of Module Designs. As presented the last five rows in Table we evaluate the impacts of the module designs in our ShareRefiner and PromptRefer. The comparison of results between the seventh and last rows demonstrates that employing the hard assignment for refining the anatomical queries can boost performances, especially for tumor segmentation. However, applying the paradigm to the other two queries (i.e., the 'Hard' column is marked v) will decrease the performance of segmentation. This decline can be attributed to the entanglement of the three query types. Such phenomenon further reinforces that anatomical and textual prompts contribute from different perspectives. Using a one-hot hard assignment ensures that each query feature remains exclusive the others, allowing queries focus on distinct visual regions without overlap. We hypothesize that anatomical queries could attend to features and distinguish between different categories based on inherent appearance characteristics. Additionally, we remove the mask mechanism in the PromptRefer and utilize a vanilla cross-attention layer to update Q', As illustrated in Table 3 this variant leads reduced performance in tumors that invade other organs (c.g., Stage- IV). The result verifies that guiding queries to focus on relevant objects is an effective strategy for achieving more robust segmentation in medical scenarios.
Visualizations. We select nine categories (five organs and four tumors) and use t-SNE (63 to
visualize the distribution of prompt features (E and Ex) and updated prompts queries (Q' and Q'r) in Figure 4 Our findings indicate that the initial prompt features are not well-separated in the feature space. For instance, the anatomical features of the right kidney and left kidney (categories 1 and 2) are entangled. Therefore, different refined paradigms need to be applied to update both queries simultancously. With our ShareRefiner, prompt features are separated in the feature space. Moreover, we can observe that the distribution of refined anatomical prompt queries is more structured than textual prompts, which further verifies that the anatomical prompt is more intuitive. Contrastive alignment is utilized to further push segmentation queries to be close to the referenced prompt for segmenting the corresponding category. To validate the effectiveness, we trained without utilizing contrastive alignment. We also use t-SNE to visualize the distribution of decoded segmentation query features Os in the right part of Figure We can observe that segmentation queries are more separated in the feature space with contrastive alignment. Additionally, we visualize the heatmaps derived from experiments in the second and third rows [Tale As shown in Figure 5 solely using textual prompts fails to cover all regions, while only relying anatomical prompts results in a high

---

Znom:In
Dual-Frompi
Arutamical
Textual
Figure 5: Heatmaps of two samples for analyzing the effectiveness of two prompts.
false positive rate. These visual examples underscore the importance of combining both types of prompts to guide medical image segmentation effectively.
5 Conclusion and Discussion
Conclusion. We present CAT, a promising attempt towards comprehensive medical segmentation via coordinating anatoul prompts. Apart from performing generic organ segmentation, CAT can identify varying tumors without human interaction. To effectively integrate two prompt modalities into a single model, we design ShareRefiner to refine latent prompt queries with different strategies and introduce PromptRefer with specific attention masks to assign prompts segmentation queries for mask prediction. Extensive experiments indicate that the coordination of these two prompt modalities yields competitive performance organ and tumor segmentation benchmarks. Further studies revealed the robust gencralization capabilities to segment tumors in different cancer stages. We hope our early exploration of the complementary advantages between anatomical and textual prompts could bring new insights into the field of the community.
Limitation and Impact. We hope our model can support professionals in the arduous clinical diagnosis process. Despite the integration of anatomical and textual prompts showing competitive performance in the segmentation tasks, the lack of general anatomical prompts encoders raises challenges, as indicated by the messy distribution of initial anatomical embeddings. Therefore, further research into improving the CT foundation models is essential. Moreover, there may also be mistakes in the segmentation results, especially when test sample contains rare types of lesions undergoes radical resection surgeries that cause large variations in the anatomical structures. Therefore, before integrating these Al-based algorithms into clinical practice, legislation needs to be developed and implemented to ensure that there are clear guidelines and standards for their use.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (No. 62301311) and the Shanghai Municipal Commission of Economy and Informatization (No. 204694).
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamnrr.. Advances in neural information processing systems, 33:18777-99 2020.
[2] Jason Wei. Xuezhi Wang. Dale Schuurmans, Maarten Bosma. Fei Xia, Ed Chi, Quoc Le, Denny Zhou, Chain-of -tthuuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -223277 2022.
[3] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2033:22003008774 2023.
10

---

[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230,,09288 2023.
[5] Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean- -ran-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint ar 11805, 2023.
[6] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang. Linjie Li, Lijuan Wang, and Jianfeng Gao. Multi- modal foundation models: From specialists to general-purpose assistants, arXiv preprint arXiv:2309. 10020, 12))2, 2023.
[7] Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Chunyuan Li, Jianwei Yang, et al. Visual in-context prompting, arXiv preprint arXiv:2311. 2023.
[8] Ehab AlBadawy. Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation brain tumors: Impact of tross-institutioaal training and testing. Medical physics, (5(3):1500-1188 2018.
[9] Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging, 32:582 -9. 2019.
[10] Shaoting Zhang and Dimitris Metaxas. On the challenges and perspectives of foundation models for medical image analysis. Medical Image Analysis, page 102996, 2023.
[11] Ziheng Zhao. Yao Zhang, Chaoyi Wu, Xiaoman Zhang. Ya Zhang, Yanfeng Wang, and Weidi Xie. One model to rule them all: Towards universal segmentation for medical images with text prompts. arXiv preprint 111::2312.17183, 2023.
[12] Yankai Jiang. Zhongzhen Huang. Rongzhao Zhang, Xiaofan Zhang, and Shaoting Zhang. Zept: Zero-shot pan-tumor segmentation via query disenytdiennnnginngg and self peflprrompiing arXiv preprint wrXiv2231.04496. 2023.
[13] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou, Clip-driven universal model for organ segmentation and tumor detection. arXiv preprint arXiv:2301.0078: 2023.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR, 2021.
[15] Jacob Devlin. Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018
[16] Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21438-21451, 2023.
[17] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
[18] Cheng Chen, Juzheng Miao, Dufan Wu, Zhiling Yan, Sekeun Kim, Jiang Hu, Aoxiao Zhong, Zhengliang Liu, Lichao Sun. Xiang Li, et al. Ma-sam: Modallty-agnosti sam adaptation for 3d medical image segmentation. arXiv preprint arXiv:2309.0888 08842, 2023.
[19] Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang. Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sam-med2d. arXiv preprint arXiv:2308. /6/84, 2023.
[20] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Am Heng, and Qi Dou. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306. 13465, 2023.
[21] Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation. arXiv preprint 23l.:2304.13885. 2023.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
11

---

[23] National Cancer Institute. Cancer staging. 2022. Accessed: 2024-05-21.
[24] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang. Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
[25] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Hcung- Yeung Shum. Mask dino: Towards a unified transformer-base framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3041-3050. 2023.
[26] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299 2022.
[27] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Lang-Chich Chen. Max-docplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVT conference computer vision and pattern recognition, pages 5463-5474, 2021.
[28] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocahulary object detection via vision and language knowledge distillation. arXiv preprint 100i 13321, 2021.
[29] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, ct al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint urXiv:2303.05499 2023.
[30] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[31] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020-1031, 2023.
[32] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11583- 11592, 2022.
[33] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu, Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[34] Golnaz Ghiasi, Xiuye Ga, Yin Cui, and Tsung-Yi Lin. Scaling open-vacahulary image segmentation with image-level lahels. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
[35] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for poccbuocay semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pages 736-753. Springer, 2022.
[36] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, al. Freeseg: Unified, universal and open-vocabulayy image segmentation. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pages 19446-19455, 2023.
[37] Shiyi Du, Xiaosong Wang, Yongyi Lu, Yuyin Zhou, Shaoting Zhang, Alan Yuille, Kang Li, and Zongwei Zhou, Boosting dermatoscopic lesion segmentation via diffusion models with visual and textual prompts. arXiv preprint arXiv:2310.02996 2023.
[38] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.09685. 2021.
[39] Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint 1xXiv:2311.3336 2023.
[40] Haoyu Wang. Sizheng Guo, Jin Ye. Zhongying Deng Junlong Cheng. Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, et al. Sam-med3d. arXiv preprint arXiv:2310.1516,, 2023.
12

---

[41] Heng Guo, Jianfeng Zhang, Jiaxing Huang, Tony CW Mok. Dazhou Guo, Ke Yan, Le Lu, Dakai Jin, and Minfeng Xu. Towards a comprehensive, efficient and promptable anatomic structure segmentation model using 3d whole-body et scans. arXiv preprint urXiv:2403. 00044003...3... 15063, 2024.
[42] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems. 34:17864 78441777777775 2021.
[43] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:190.003323. 2019.
[44] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softtax. In ICLR, 2017.
[45] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. ICLR, 2017.
[46] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information procssing systems, 30, 2017.
[47] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[48] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In CVPR, pages 20730-20740, 2022.
[49] OpenAI. Gpt-4 technical report, 2023.
[50] Bennett Landman, Zhoubing Xu, Igelsias, Martin Styner, T Langerak, and Amo Klein. Miecai multi-atlas labeling beyond the cranial vault- workshop and challenge. In Proc. MICCAI Multi-Arlas Labeling Beyond Cranial Works-Wrksshop Challenge, volume 5, page 12, 2015.
[51] Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in Neural Information Processing Systems, 35:36722-36732 2022.
[52] A Emre Kavur, N Sinem Gezer, Mustafa Banş, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Due Duy Pham, Soumick Chatterjee, Philipp Frnst, Savaş Özkan, et al. Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis, 69: 1019500 2021.
[53] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L. Rubin. Ct-org, a new dataset for multiple organ segmentation computed tomography. Scientific Data, 7(1):381, 2020.
[54] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, al. Abdomenct-Ik: Is abdominal organ segmentation solved problem. IEEE TPAMI, 2021.
[55] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan; Multi-level deep convolutional networks for automated pancreas segmentation. In Medical Image Computing and Computer- Assisted Intervention-MICCN. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 556-564. Springer, 2015.
[56] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song, Xiaofan Zhang, Kang Li, Dimitris N Metaxas, Guotai Wang, and Shaoting Zhang. Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from et image. Medical Image Analysis, 82: 12:102642, 2022.
[57] Patrick Bilic. Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis. Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84:102680, 2023.
[58] Nicholas Heller, Sean McSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Joel Rosenberg, et al. An international challenge to use artificial intelligence to define the state-of sfate-of-the-art the-art in kidney and kidney tumor segmentation in ct imaging., 2020.
[59] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp- Hopp-Schneider, Bennett A Landman, Geert Litjens, Bjoemn Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022.
13

---

[60] Jun Ma. Yao Zhang, Song Gu, Cheng Ge. Shihao Ma. Adamo Young, Cheng Zhu, Kangkang Meng. Xin Yang. Ziyan Huang. Fan Zhang, Wentao Liu, YuanKe Pan, Shoujin Huang, Jiacheng Wang, Mingze Sun, Weixin Xu, Dengqiang Jia, Jae Won Choi, Natália Alves, Bram de Wilde, Gregor Koehler, Yajun Wu. Manuel Wiesenfarth, Qiongjie Zhu, Guoqiang Dong, Jian Hc, the FLARE Challenge Consortium, and Bo Wang. Unlcashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge. arXiv preprint arXiv:2308..0380 2023.
[61] Ali Hatamizadch, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In International MICCAI Brainlesion Workshop, pages 272-284. Springer, 2021.
[62] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier Hein. nnu-net: self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):003221, 2021.
[63] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008.
[64] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 171.5171 2017.
arXiv preprint
14

---

A Appendix supplemental material
A.1 Textual Prompts Construction
The instruction for GPT-4
Short phrase templates
Please describe provided organs and tumors. Descriptions should include the location, ita anatonical atructure, common pathological findinga, and counon tunors (for organs), Anavers must be based observations from CT imaging.
computerized tomography [CLS].
A photo of a [CLS].
There is [CLS] in this computerized tomegraphy,
[CLSS]
Table 4: The strategy of constructing textual prompts.
We call GPT-4 to generate long descriptions for each category and use templates to construct some short textual prompts, as shown in A.1 We the long description for the positive category and randomly sample short phrases for those negative categories. Positive categories indicate labeled categories in the case.
A.2 Dataset Details
The overall categories used in our paper consist 23 organs and 6 anomalies. The test set of MSD the same as the official code of [13). Details of utilized datasets are shown as follows:
(1) The BTCV dataset [50] consists of 30 abdominal CT scans from 13 annotated organs, conducted under the supervision of radiologists at Vanderbilt University Medical Center.
(2) The CT-ORG dataset [53] encompasses 140 CT images, featuring six organ classes. These images primarily display liver lesions, including both benign and malignant types.
(3) The AbdomenCT-IK dataset [54] comprises 1,112 CT scans pooled from five datasets, annotated for the liver, kidney, spleen, and pancreas.
(4) CHAOS [52) offers 40 CT scans of healthy abdominal organs for multi-organ segmentation, specifically excluding any pathological abnormalities such as tumors or metastasis.
(5) AMOS22 [5T]. multi-modality abdominal multi-organ segmentation challenge of 2022, includes
500 CT scans with voxel-level annotations of 15 abdominal organs.
(6) The WORD dataset [56] collects 150 CT scans from patients prior to radiation therapy single center, with each volume consisting of 159 to 330 slices and comprehensively annotated for 16 anatomical organs.
(7) Pancreas-CT (55] contains 82 contrast-enhanced abdominal CT volumes focused solely on the pancreas, annotated by an experienced radiologist and excluding any pancreatic tumors.
(8) The LiTS dataset [57] comprises 201 contrast-enhanced abdominal CT scans (131 for training and 70 for testing), acquired across six clinical sites using various scanners and protocols, with resolution range from 0.55 to 1.0 mm and slice spacing from 0.45 to 6.0 mm.
(9) KITS 58 includes 300 CT scans (210 for training and 90 for testing) with annotations provided by the University of Minnesota Medical Center, each featuring one or more kidney tumors.
(10) The Medical Segmentation Decathlon (MSD) (59 comprises 947 CT scans targeting liver, lung, pancreas, colon, hepatic vessels, and spleen, encompassing a total of four organs and five tumors.
(11) Our in-house dataset contains 80 CT scans of patients diagnosed with colon cancer, annotated by an experienced gustroenterologistt and verified by a senior radiologist. All scans maintain consistent in-plane dimension of 512 x 512 pixels, while the z-axis dimension ranges from 36 to 146, with a median of 91. Note that there are some cases where the cancer stage not be indentified.
A.3 Implementation Details
AdamW optimizer 64 is utilized as the optimizer. The default initial learning rate is 10-4, and we decay it following the learning rate scheduling strategy of [61]. Each volume is cropped into patches with a size of 96 x 96 x 96. Random shift, zoom and scale, and scaling are applied on-the-fly to improve the generalization. Our framework is implemented using Py Torch and all experiments are
15

---

Datasets
#SScans
Annotated categories
BTCV 50
30
Spi, RKid, LKid, Gall, Fso, Liv, Sto, Anr, IVC, R&EVViiss, Pan, RAG, LAG
CT-ORG 5
140
Lung, Liver, Kidneys and Bladder
AbdomenCT-IKK 54
1000
Spicen, Kidney, Liver, Panereas
CHAOS 12
40
Liver, Left Kidney, Right Kidney, Spleen
AMOS22
500
Spl. RKid. LKid, Gall, Eso, Liv, Sto, Aor. IVC, Pan, RAG, LAG, Duo, Bla, Pro/UTE
WORD
150
Spl. RKid, LKid, Gall, Eso, Liv, Sto, Pan. RAG, Duo, Col, Int, Rec, Bla, LFH. RFH
Pancreas-
55
Pancreas
LITS
201
Liver, Liver Tumor
KiTS
58
300
Kidney, Kidney Tumor
Spl, Liver, Liver Timor, Lang Tumor, Colon Tumor, Pancreas and Pancreas Tumor, Hepatic Vessel and Hepatic Vessel Tumor
MSD CT Tasks
[59
947
In in-house dataset
80
Colon Tumor with four subtypes
Table 5: The information for all datasets.
conducted on 8 NVIDIA A100 GPUs. For the evaluation, the Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95) are utilized to gauge the performance of organ/tumor segmentation.
A.4
Results Details
Table 6: Organ segmentation performance on FLARE22. The results( (%) are evaluated by DSC.
Scores of SAM-based are adopted from the CT-SAM-Med3D 41]. f denotes obtained via the official pre-trained weights. * means implemented from the official code and trained on the same dataset. Abbreviations: "Liv."-Liver, "R_Kid." K.d."k-i"glllll"" Kidney, "Spl."-Spleen. "Pan." -Pancr-Paa "Aor."
Aorta, "TVC"-Inferin Vena Cava, "RAG" -Right Adrenal Gland, "LAG" -eft Left Adrenal Gland, "Gal." Gallbladder, "Eso."-Esoppie EEsohagus, "Sto." "ito."-Stonn -Stomach, "Duo." "auo."-uucc -Duodenum, __d. I.dd.iio Vena Cava.
Spt.
LAG
Sto.
33.9.15
35.2
52.6.112
99.1.me
42.51393
48.7-126
6477.9
$4.0.13
89.7159
SAM-MalD
SAM MeBD 30
854
84.7.n
60. 60.41397
44.5.114
35.3.183
32.6.144
46,9.195
8491s9
[9)
71.71302
7591225
83.111211
50.3.139
42.11153
49.7-125
75.1-22
CT SAMID
95.6120
95.0115
96.1164
94.5.33
91.8147
78.4158
82.5149
8K4181
82.9.161
92.3144
73.21365
94.114
Universal
96.41145
13:4-145
96,9115
ZePT"
96.7120
90.0.9
67.2124
66.3.181
74.22n5
63,2.113
59.11ms
97.2110
91.7.13
96.3141
97.1.11
89.2176
90.3991
$3.0129
73.61833
74.3.109
58.2:133
734.33
97.3187
Table 7: Segmentation performance of tumors on MSD [59] and In-house dataset. We compare our method with traditional and promptable methods. denotes obtained via the official pre pre-trained weights. * means implemented from the official code and trained on the same dataset.
Abdomen)
In-house (Colon Tumor)
Method
Läver
Pancreas
Hepatic Vessel
Colon
T1
T2
T3
T4
Avg.
nnUNNet
66.42.aws
43.50_
66.90
41.41
19.5
45.061199
44.87_1124
43.00_.11
Swin UNETR"
48
68.67=231
41,77=329
63 63.32=253
39.35
21.40223
33.52+304
46.11=253
52.72=22.
45.92-265
SAM-Med3D
44.78:337
40.05 2183
84.862263
39 1
34,28a354
42.654191
50,20-169
42.65
17.1
SegVall 38
26.20
46.36_72x
68.57lmk2
60.63
42.63mmx
60.17_
49.83,am1
50.28..167
Universal
13
65.68=272
45.72=331
66.31=215
42 262304
7.11-22
43.28 23.288200
46.52=236
$3.08
47.14=29.
7ePT'
12
68.58
44.39
68.124006
40.38,299
23.87
34.641ma
50.81
51,09
46.28_18.5
CAT
72.73 2x3
49.67 .937
70.1
18.31
30.62
45.61
$5.85 217
$7.37 2244
53.35 17.8
Further results are depicted in Figure 3 Our model demonstrates the capability to segment all structures of the aorta, including arcus aortae. This comprehensive segmentation advantageous in worl-world scenarios, although it slightly lowers the Dice score.
16

---

Figure 6: Visualization of segmentation result of CAT. The left part is the ground truth and the right part is ours. The first five rows are results of organ segmentation results, and the others are tumor segmentation results. From the top down are liver tumors, pancreas tumors, Hepatic Vessel tumors, colon tumors, and colon tumors in T4. For each tumor type, we present two cases.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction of the paper accurately reflect the contributions and scope of the research documented. Both sections clearly articulate the theoretical and experimental advancements achieved, aligning with the results discussed the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? [Yes]
Answer:
Justification:
The paper discusses the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper is based solid experimental results and does not involve theoreti- cal outcomes.
Guidelines:
The answer NA means that the paper does not include theoretical results.
18

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The information needed to reproduce the main experimental results is intro-
duced in Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Codes and data are available at supplemental materials. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
19

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http://gggiiee/ooeeuuuiiiieeeee / //nips.cc. wwtpcccwccgguupiiaeeeeeeeooppppe des/ CoodeSubiiiss settuubmissiooooo seinSubbbiiioooloiioy onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details can be seen in Section 4 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports the Standard Deviation in Table 6 Table 7 Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
20

---

Justification: Sufficient information on the computer resources can be seen in Section A.3 Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuripst httppp///aaarrssscc/////////iica csesideliessse
Answer: [Yes]
Justification: Authors have reviewed the NeurlPS Code of Ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the potential positive societal impacts and negative societal impacts of the work in Section Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset. Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasetssppprsstttt package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
22

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We obtain Institutional Review Board approvals of the in-house dataset and there is no potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "UFHtqk7dDo", "reviewer": "Reviewer_wEKt", "summary": "This paper proposed CAT, a promptable segmentation model that utilizes the strengths of both visual and textual prompts without human interaction, aiming at a fully automatic model for medical professionals. Extensive experiments demonstrate the benefits of coordinating anatomical prompts and textual within one model. CAT achieves state-of-the-art performance on multiple segmentation tasks and has generalization capability to diverse tumor types.", "questions": "1. What do the textual prompt and visual prompt look like? It is suggested that an example be given in the Supplementary. \nIf CAM is only trained on one dataset, will it surpass nnUNet and other comparisons?\n2. How large are CAT and those comparison models? It is suggested to report the number of parameters.\n3. It is suggested to give a concrete example that how visual and text prompt improve the segmentation.", "limitations": "The paper proposed in the introduction that medical data is challenging because of the long-trailed problem but did not illustrate how CAT helps solve this problem.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is OK. CAT combines text and visual prompts, which could be needed in clinical scenarios. \n2. The experiment CAT proves that combining visual and textual prompts is essential.\n3. The experiment shows that CAT can deal with challenging small region segmentation and tumor segmentation.\n4. CAT applied domain knowledge generated from GPT4, which is innovative. And one board-certified physician is recruited to check the text prompts.", "weaknesses": "1. It is not clear how those comparison methods were trained. Were they also trained on the same 10 datasets as CAT? For example, how was nnUNet trained and tested (since the number output channel of nnUNet is fixed)? \n2. The writing is a bit confusing. For example, what is the backbone of ShareRefiner and PromptRefer? The similarity matrices need some mathematical illustration.\n3. It is confusing that Table 3 has the same setting in the last two rows. Is that a typo?\n4. The paper did not mention what pre-trained model is used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We will address your concerns in the following parts.\n# W1: Details of Experiments\nSorry for the confusion caused by the omission of certain experimental details. In our experiments, we followed Universal's experimental settings [1]. We also trained the comparison methods that we implemented ourselves on the same 10 datasets. Specifically, we modified the number of output channels in the baseline models (e.g., nnUNet, Swin UNETR) to enable them to perform multi-organ and tumor segmentation.\n# W2: Details of ShareRefiner and PromptRefer\nWe hope the provided illustration will give you a clearer understanding of our ShareRefiner and PromptRefer. Both ShareRefiner and PromptRefer are built upon the attention mechanism [2].  Specifically, the ShareRefiner consists of a series of cross-attention blocks where each type of query (i.e., segmentation queries $\\mathbf{Q}_S$ , anatomical prompt queries $\\mathbf{Q}_A$ and textual prompts queries $\\mathbf{Q}_T$) performs cross-attention with the extracted visual features. We use cross-attention to assign all possible relevant visual features to textual prompt queries, and use hard assignment for anatomical prompt queries. The reason for employing hard cross-attention is to ensure that each anatomical query gathers discriminative visual regions without overlaps. The effectiveness of the hard assignment is verified by the results in the following table. \n|Tumor Dice(%)|Liver|Pancreas|Hepatic Vessel|Colon|Colon in T4|\n|-----------------------------|:---:|:------:|:------------:|:---:|:---------:|\n|CAT w/othe hard assignment|72.18|46.46|69.97|46.65|58.49|\n|CAT w/othe PromptRefer mask |72.64|48.49|69.02|47.29|53.67|\n|CAT|72.73|49.67|70.11|48.31|57.37|\n\nIn the PromptRefer, refined segmentation queries $\\mathbf{Q'}_S$ engage in cross-attention with refined anatomical prompts queries $\\mathbf{Q'}_A$ and textual prompts queries $\\mathbf{Q'}_T$  to enhance segmentation. We employ a conventional attention mechanism, supplemented by carefully crafted attention masks, These masks force a group of prompt queries is employed to a specific segmentation query. This process aligns with empirical insights suggesting that accurately localizing typical tumors necessitates recognizing anomalous features within the pertinent organ. As can be seen from the table, this strategy helps to segment tumors that invade other organs (e.g., Stage-IV). We hope the above explanation and the provided code in Supplementary Material can address your confusion.\n\n# W3&W4: Clarification of Table 3 and Pre-trained models\n1. Sorry for the confusion caused by the use of symbols (✓ and ✔️). As discussed in Section 4.2, the second-to-last row in Table 3, marked with a ✔️, indicates our use of hard assignment across all cross-attention layers. We conducted this experiment to validate our hypothesis that different types of prompts play distinct roles in aggregating visual features. We will clarify the symbol in the revised caption of Table 3. \n2. The pre-trained model employed for anatomical prompts is the Swin UNETR. We utilize Clinical-Bert to encode textual prompts.\n# Questions:\nWe hope the following responses will address your questions:\n\n**Q1&Q2**: We present examples of visual prompt in **Figure** (Figure 1 of the rebuttal PDF) and the details of textual prompt in **[Json](https://anonymous.4open.science/r/Reb/textual_prompts.json)**.\nWe have added more comparisons in the case of training models on the dataset and report the number of parameters of CAT and those comparison models. The results are shown in the following tables. \n\n|MSD Tumor (Dice (%))|Liver|Pancreas|Hepatic Vessel|Colon|\n|--------------------| :----------: | :-------------: | :-------------------: | :----------: |\n|nnUNet |64.52 |43.80| 64.62 |40.44|\n|ZePT|66.53 |44.05| 66.18 |40.16|\n|CAT|**69.65** |**47.55**|**69.43**|**46.13** |\n\nWe trained the CAT, ZePT, and nnUNet models on four MSD datasets. The results demonstrate that CAT still achieves superior outcomes even when trained on a single dataset, further validating our approach of integrating textual and visual prompts for segmentation. \n\n|Model|CAT|nnUNet|SAM-Med3D|SegVol|ZePT|\n|----------|:-------:|:-------:|:---------:|:-------:|:-------:|\n|Parameters|345.53M|235.02M|374.42M|673.86M|745.94M|\n\n**Q3**: In Figure 4 of our paper, we illustrate how visual and textual prompts enhance segmentation. Unfortunately, the heatmap format may have caused some misunderstanding. To address this, we provide a comparison in **Figure** (Figure 2 of the rebuttal PDF). While our textual prompts cover most scenarios, using them alone fails to encompass all regions. This observation further supports the claim that aligning textual and visual representations is challenging. Conversely, relying solely on anatomical prompts results in a high false positive rate and overly sharp boundaries. We hope these visual examples provide clearer clarification for you. \n\n# Limitations\nWe appreciate your suggestions. To assess CAT’s efficacy in dealing with rare cases (i.e., the long-tailed problem), we introduce an in-house dataset where colon tumors have invaded adjacent organs. According to medical literature [3] [4],  T4 colorectal tumors, which represent only about 5-8.8% of colon tumor cases, pose significant challenges in diagnosis and treatment. Our results demonstrate that CAT significantly outperforms other models in segmenting T4 colon tumors, underscoring the effectiveness of our design in handling complex medical scenarios. We hope our early exploration could bring new insights into the field of the community.\n\n# References\n[1]  Clip-driven universal model for organ segmentation and tumor detection.\n\n[2]  Attention is all you need.\n\n[3]  Results after multi-visceral resections of locally advanced colorectal cancers: an analysis on clinical and pathological t4 tumors. \n\n[4]  Identification of risk factors for lymph node metastasis of colorectal cancer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IuZfg2sk2", "reviewer": "Reviewer_qosx", "summary": "The paper \"CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation\" introduces a novel dual-prompt schema that leverages both anatomical and textual prompts for medical image segmentation. The proposed CAT model coordinates 3D anatomical prompts with enriched textual prompts to improve segmentation accuracy for various organs and tumors. Key contributions include the development of the ShareRefiner and PromptRefer modules to refine and integrate these prompts, resulting in superior performance on multiple public CT datasets and an in-house dataset. The approach demonstrates enhanced generalization capabilities and robustness in complex medical imaging scenarios.", "questions": "1. In equation 3, why is a Linear function used to convert EAE_AEA​ to QAQ_AQA​? There is no explanation provided for this choice. I assume it is to reduce the dimensions of the embedding space, but clarification is needed.\n\n2. There should be more explanations about learnable segmentation queries. For example, how are they initialized? What is their purpose? Beyond their use in architecture, a high-level explanation of how they help predict masks would aid readers in understanding and relating to the motivation behind these techniques.\n\n3. Why do some models present HD95 values while others do not? The current reasoning isn't sufficient.\n\n4. Figure 1, some modules have the \"snowflake\" icons, which the authors don't explain why. Assuming these show that these are frozen models that have not been trained, do they mean any modules that don't have these are all trained?", "limitations": "Please refer to the Weaknesses section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel approach by combining anatomical and textual prompts, leveraging the strengths of both to enhance medical image segmentation.\n\n2. The development of the ShareRefiner and PromptRefer modules demonstrates a sophisticated method for refining and integrating multimodal prompts, leading to improved segmentation accuracy.\n\n3. The paper provides a thorough experimental analysis, including ablation studies and qualitative comparisons, to substantiate the effectiveness of the proposed methods.\n\n4. The work focuses on both organ and tumor segmentation, which is commendable and proves the model's performance overall.", "weaknesses": "1. The motivation for using textual descriptions was to provide the model with specific knowledge about each image, including details such as cancer stages and density. However, the authors use only fixed generic textual information generated by a language model in this work. This approach does not fully capture the intended motivation. If the textual information included specific details for each sample, demonstrating how changes in staging affect segmentation performance, it would better support their claim about the utility of textual information.\n\n2. I am not convinced of CAT's superiority in the comparative results. For example, CT-SAM3D seems to perform better in organ segmentation in most cases, and where CAT excels, the improvement is minimal compared to other text-based segmentation models. Additionally, why aren’t SOTA segmentation models like UNet, nn-UNet, and Swin UNETR, etc included for organ segmentation, and vice versa for tumor segmentation? Only comparing with SAM-based models for organ segmentation is insufficient, especially since SAM is known to have poor performance in medical image segmentation. \n\n3. The authors claim that CAT performs better on the in-house dataset, especially for tumors at different stages. However, the textual prompts used by CAT are very generic and do not include the cancer stage information. Thus, the reasons for CAT's superior performance are unclear.\n\n4. The paper lacks sufficient novelty, as it combines already existing methods, such as integrating text and visual prompts, where text is intended to provide rich semantic information. However, it does not adequately explain how the text contributes to performance improvement. This work appears to be an incremental extension of ZePT, with only the creation of visual prompts distinguishing it from similar efforts.\n\n5. Exactly how contrastive alignment works in PromptRefer isn't clear. This should be further enhanced.\n\n6.  Many of the techniques are described in terms of their usage, but the underlying motivation for their utilization is not clearly articulated. For instance, the rationale for choosing hard assignments in ShareRefiner is not explained. If the intention is to follow ZePT, this choice is questionable because ZePT uses hard assignments to distinguish between healthy organs and tumors in feature space. The motivation behind these decisions is lacking.\n\n7. Figure 2 (c) should have more explanations. It is not clear right now.\n\nOverall, the authors should place greater emphasis on the motivations behind the chosen techniques. Currently, it appears they are following these methods simply because they work, which is not a robust standard. Clear justification for each technique would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. Below, we will address your concerns on each point.\n# W1&W3: Details of textual descriptions and Reasons for superior performance\nAs detailed in Section 3.4, for each category, we curate long descriptions. As highlighted in the motivation section, learning the alignments between textual and visual representations is challenging, particularly for fine-grained details. In real-world scenarios, obtaining specific details for each sample prior to segmentation is impractical. Considering the vast diversity of tumors, we leverage general knowledge to convey the concept of each tumor type. Details can be in **[Textual Prompts](https://anonymous.4open.science/r/Reb/textual_prompts.json)**. To better support our claim about the utility of textual prompts, we conducted ablation studies in Table 3 of our paper. The observed improvement in the third row (compared to the first row) underscores the effectiveness of textual prompts in helping deal with the diversity and variance of tumors.\n\nAs shown in Table 2, CAT shows strong capabilities in dealing with tumors at different stages. The reasons can be attributed to several factors: 1. The textual prompt for colon tumors provides context for '**surrounding inflammation, adjacent organ invasion, or regional lymph node enlargement**,' which are critical in different stages. 2. The visual prompts offer intuitive and direct examples to highlight the appearance features.  3. The carefully designed mask in the PromptRefer helps to handle tumors that invade nearby organs or tissues.\n# W2: Organ segmentation results\n1. The detailed organ-wise segmentation results are derived from the original CT-SAM-Med3D paper. It is important to note that the results presented were achieved with a prompt number ($N=5$), which is derived from the ground truth. As illustrated in the cited **Figure** (Figure 3 of the rebuttal PDF) from CT-SAM-Med3D, reducing the number of prompts from 5 to 1 leads to a significant performance decrease, with the average Dice score falling below 85%. \n\n2. The average organ segmentation scores for nn-UNet and Swin UNETR are presented below. Our design enables CAT to outperform most SAM-based models in the medical domain and established state-of-the-art (SOTA) segmentation models.\n\n- **nn-UNet**: Pan.-79.36, RAG.-68.37, LAG.-73.31, Eso.-76.16, Duo.-72.43, **Avg.-83.38**\n- **Swin UNETR**: Pan.-87.05, RAG.-66.17, LAG.-74.18, Eso.-75.65, Duo.-48.51, **Avg.-82.78**\n\n# W4&W6&W7: Details of our method\nFactually, CAT significantly diverges from ZePT in several key aspects. Besides the introduction of visual prompts, CAT utilizes more descriptive texts as textual prompts to convey general concepts, while ZePT only uses knowledge for the final alignment. For handling two types of prompts, CAT employs distinct strategies: we use soft assignment to gather all potentially relevant visual features for textual prompts while applying hard assignment to secure discriminative visual regions without overlaps.  This is markedly different from ZePT’s unified refinement process. Furthermore, in PromptRefer, we carefully design attention masks according to clinical knowledge. In summary, CAT mainly focuses on combining anatomical and textual prompts to enhance segmentation tasks, whereas ZePT delves into identifying anomalies from original visual features.\n\nFigure 2(c) illustrates how segmentation queries interact with a mixed group of refined prompt queries via a cross-attention mechanism in the PromptRefer module. For example, Stage-IV colon tumors often invade adjacent organs such as the intestines. In such cases, the segmentation query $\\mathbf{Q'}_{Si}$ for colon tumors is specifically directed to attend to the refined query set of anatomical $\\mathbf{Q'}_A$ and $\\mathbf{Q'}_T$ encompassing Colon, Stomach, Duodenum, Intestine, and Rectum. We hope the **Figure** (Figure 4 of the rebuttal PDF) clarifies you more clearly. \n# W5: Effectiveness of contrastive alignment\nContrastive alignment is utilized to further push segmentation queries to be close to the referenced prompt for segmenting the corresponding category. To validate the effectiveness, we trained without utilizing contrastive alignment. The results are shown in the following table. Eliminating contrastive alignment leads to a performance drop.\n\n|Dice(%)|Pan.|RAG.|LAG.|Eso.|Duo.|Liv. Tumor|Pan. Tumor|HV. Tumor|Colon Tumor|Colon Tumor(T4) |\n|:-----------------------------|:------:|:------:|:------:|:-------:|:------:|:--------------:|:-----------------:|:-----------------------:|:--------------:|:-------------------:|\n|CAT w/o  contrastive alignment|87.97|73.61|72.45|79.06|71.35|71.36|47.61|68.10|46.50|56.07|\n|CAT|89.24|73.69|74.63|80.10|73.46|72.73|49.67|70.11|48.31|57.37|\n\nWe also use t-SNE to visualize the distribution of decoded segmentation query features $\\mathbf{O}_S$ in **Figure** (Figure 5 of the rebuttal PDF).  We can observe that segmentation queries are more separated in the feature space with contrastive alignment.\n# Questions:\nThanks for your careful reading. As you mentioned, the Linear function is to transform the dimensions of the embedding space. The \"snowflake\" icons in Figure 2 indicate that these are frozen models that have not been trained. Following previous work, the learnable segmentation queries are initialized randomly. Each of them is responsible for segmenting one category.\n\nThe reason for not reporting HD95 scores for SAM-based methods: The HD95 metric is used to assess the accuracy of image segmentation by measuring the largest distances between predicted and actual segmentations. When applying SAM-based methods that require predefined target key points as the prompts for segmentation, the predefined keypoints could artificially enhance segmentation accuracy near those points. Hence, reporting HD95 scores for SAM-based methods against others without such provisions is considered inequitable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7HEE7wOJ6", "reviewer": "Reviewer_Yp5t", "summary": "This paper proposes a prompt-based deep model for organ and tumor segmentation. The authors leverage two types of prompts—cropped target volumes and textual descriptions—to perform the segmentation. The proposed model demonstrates good performance across three datasets for the segmentation task.", "questions": "See the above weaknesses.", "limitations": "The authors have analyzed the limitations of this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Point out several critical challenges in medical image segmentation, such as the long-tailed nature, variations in shape, size, and density distribution, and blurring boundaries.\n2. Utilize both textual and visual prompts for medical image segmentation, achieving good performance.", "weaknesses": "1. The motivation is not convincing. Although the authors highlight several critical issues, it remains unclear how the two types of prompts address or refine them. For instance, textual descriptions lack detailed and quantitative measurements. How can they denote invading target boundaries? How can the effectiveness of terms like 'greater' or 'deeper' be evaluated?\n\n2. Several experimental settings are not standard. For example, in the Anatomical Prompt Encoder, the input sizes are the same. How does this account for small or large targets?\n\n3. Figure 2 is unclear. What is the output? During inference, do both prompt inputs need to be used?\n\n4. For comparisons, the authors trained the model on 10 public datasets. However, only ZePT, nnUNet, and Swin UNETR were trained in the same setting. Therefore, the conclusions drawn are not reasonable. Besides, in Table 1, CT-SAM3D achieves the best performance.\n\n5. The overall organization is unsatisfactory. First, as mentioned, the motivations could be improved to better align with the technical designs. Second, despite the better performance, no new insights are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive suggestions. Below is our detailed response to answer your concerns.\n# W1&W5: Motivation and Details of textual prompts \nThe motivation for using textual descriptions was to provide the model with the general concept of each category. While descriptive texts encompass intricate and rare anomalies using domain knowledge, learning alignments between textual and visual representations remains challenging, particularly for fine-grained details such as tumors with variations in shape, size, density distribution, and blurred boundaries. In real-world scenarios, acquiring specific details like quantitative measurements for each sample prior to segmentation is often impractical. Considering the vast diversity of tumors, we utilize general knowledge to convey the concept of each tumor type. Given the vast diversity of tumors, we employ general knowledge to convey each tumor type's concept. Utilizing GPT-4, we generated descriptions with medical domain knowledge for each category over 20 times. A board-certified physician then refined these descriptions based on the results. Further details on our textual prompts are discussed in the **[Text Descriptions](https://anonymous.4open.science/r/Reb/textual_prompts.json)**. For example, the description of colon tumors (\"On CT scans, colon tumors can present with varying densities and might be associated with surrounding inflammation, adjacent organ invasion, or regional lymph node enlargement.\") contains an abstract notion of invading target boundaries.  During the training process, we use the long description for the positive categories and randomly sample short phrases for those negative categories. In the inference stage, we apply the long description for all categories.  To verify the effectiveness of textual descriptions, we replace them with short phrases during the inference stage, the result can be seen as follows: \n\n|Dice(%)|Pan.|RAG.|LAG.|Eso.|Duo.|Liv. Tumor|Pan. Tumor|HV. Tumor|Colon Tumor|Colon Tumor(T4)|\n|:-------------------|:------:|:------:|:------:|:-------:|:------:|:--------------:|:-----------------:|:-----------------------:|:--------------:|:-------------------:|\n|CAT w. short phrases|88.87|72.17|73.12|74.37|68.99|    70.65|47.80|68.77|46.31|54.09|\n|CAT|89.24|73.69|74.63|80.10|73.46|72.73|49.67|70.11|48.31|57.37|\n\nThe observed declines demonstrate that textual descriptions, which cover the general concept of potential cases, significantly benefit the tumor segmentation process. To provide an intuitive understanding of these results, we present the qualitative results in the **Figure** (Figure 6 of the rebuttal PDF). Additionally, we enlisted a physician to annotate the tumor regions. The figure shows that a lack of detailed knowledge leads to overlooking crucial details. Importantly, the results from CAT are more closely aligned with those delineated by the expert, who possesses professional medical knowledge. This validates our approach of enhancing the segmentation process by incorporating comprehensive textual knowledge from the medical domain.\n\nWe hope the updated clarification in the **Author Rebuttal** could provide a clear understanding for you.\n\n# W2:  Details of Anatomical Prompt\nWe are very grateful for your serious reading. In our paper, we leverage the bounding box derived from masks in the public dataset and relevant anatomical structures to prepare a set of prompt volumes for each category, which we then standardize to a uniform size. The process begins by cropping the image according to the bounding box. Subsequently, we employ two different strategies to adjust the cropped volumes to the required input size of the Anatomical Prompt Encoder ($96 \\times 96 \\times 96$): 1. For volumes larger than the target size, we resize them to $96 \\times 96 \\times 96$. 2. For smaller volumes, we re-crop the image using a center-crop paradigm to achieve the size of $96 \\times 96 \\times 96$. It is important to note that all anatomical prompts used for tumor categories are processed using the second strategy. We exclude cases where the tumor size exceeds $96 \\times 96 \\times 96$, as these instances have been effectively addressed by previous methodologies. Our paper primarily focuses on the integration of anatomical and textual prompts for challenging segmentation tasks.\n# W3: Clarification of Figure 2\nSorry for any misunderstanding caused by the omission of image descriptions. As shown in the **Figure** (Figure 7 of the rebuttal PDF), the part highlighted by a red box is the output. We chose this method of illustration to provide a more intuitive understanding; we regret any confusion this may have caused. The segmentation maps are obtained by a multiplication operation between the decoded segmentation query features $\\mathbf{O}_S$ and the pixel embedding map $O$. During inference, both prompt inputs need to be used.\n# W4: Experimental Details\nSorry for any confusion caused by missing details in our experimental descriptions. In our experiments, the results for the Universal model were derived from the official pre-trained weights, which were also trained on the same datasets as our study. In contrast, the SAM-based models were trained on a broader set of datasets, including the ten datasets discussed in our paper.  Regarding the CT-SAM3D model, the detailed organ-wise segmentation results were sourced from the original CT-SAM-Med3D paper. It is important to note that the results presented were obtained using a prompt number ($N=5$), which is derived from the ground truth. As indicated in the subsequent **Figure** (Figure 3 of the rebuttal PDF), there is a significant performance drop in the average Dice score when the number of prompts decreases from 5 to 1, falling below 85%. Thanks to the design of our CAT model, it demonstrates superior results in organ segmentation without the need for human interaction", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad that reviewers are generally interested in our proposed method of combining anatomical and textual prompts for medical image segmentation. We also thank all reviewers for their insightful and constructive suggestions, which helped improve our paper further. To provide a clearer understanding, we update the backbone and motivation in our revised manuscript according to the comments. **All figures used in the rebuttal are shown in the attached PDF**.
# Backbone
**We hope that the illustration provided below will enhance the clarity and understanding of our paper.**

Previous promptable segmentation models in the medical domain can be categorized into textual-prompted methods and visual-prompted methods. Despite promising advancements, relying solely on visual or textual prompts has limitations. Textual-prompted methods utilize textual representations from referred text phrases to guide the segmentation process, requiring alignment between visual and textual representations. Although descriptive texts can cover intricate and rare anomalies with domain knowledge, data scarcity due to long-tailed distribution hinders the effective learning of alignments between textual and visual representations. This issue is particularly significant in the medical domain, where numerous corner cases (e.g., tumors with variations in shape, size, density distribution, and blurring boundaries) need to be addressed. Visual prompts, on the other hand, do not require cross-modal alignment and provide a more intuitive and direct method. However, they fail to convey the general concept of each object. For instance, tumors in different cancer stages exhibit diverse shapes and sizes, necessitating a comprehensive image collection to visually convey the abstract notion. In this work, we aim to develop a segmentation model that leverages the strengths of both visual and textual prompts without human interaction, striving for a fully automatic model for medical professionals.

Specifically, our visual prompts are derived from the relevant anatomical structures (i.e., cropped 3D CT images) and textual prompts are curated based on medical domain knowledge, as shown in the following links: **Anatomical prompts**(Figure 1 of the rebuttal PDF) and **[Textual Prompts](https://anonymous.4open.science/r/Reb/textual_prompts.json)**. For these two prompts, we apply different feature-gathering strategies. We use soft assignments to gather all possible relevant visual features for textual prompt queries and hard assignments to obtain discriminative visual regions without overlaps.
# Motivation of the Module Design
**We hope this section of our discussion clarifies the motivation behind our module design (the concerns raised by Reviewer qosx), specifically the rationale for our proposed ShareRefiner and PromptRefer.**

The underlying motivation of ShareRefiner module is to provide general concepts that encompass a wide range of scenarios within the medical domain via textual prompts and offer more intuitive and direct cues to mitigate the coarse visual-textual alignment issues via visual prompts. Consequently, we utilize soft assignment to assign all potentially relevant visual features to textual prompt queries while applying hard assignment for anatomical prompt queries to ensure that each anatomical query accurately captures discriminative visual regions without overlap. The rationale for designing attention masks in PromptRefer is to direct queries to focus specifically on relevant objects and prevent the introduction of noise from irrelevant regions. In clinical practice, accurately localizing the typical tumor requires being aware of the anomalous features in the relevant organ, and even identifying organs requires focusing on the anatomical structures involved.  This strategy aligns with practical experience, which suggests that effective segmentation of target objects necessitates a heightened focus on the relevant contextual details. 

# Contributions
1. We present a promising attempt toward comprehensive medical segmentation via coordinating anatomical-textual prompts. Apart from performing generic organ segmentation, our model can identify varying tumors without human interaction. 

2. To effectively integrate two prompt modalities into a single model, we design ShareRefiner to refine latent prompt queries with different strategies and introduce PromptRefer with specific attention masks to assign prompts to segmentation queries for mask prediction. 

3. Extensive experiments indicate that the coordination of these two prompt modalities yields competitive performance on organ and tumor segmentation benchmarks. Further studies revealed robust generalization capabilities to segment tumors in different cancer stages.  

4. We highlight several critical challenges in medical image segmentation and call for further research on utilizing both textual and visual prompts to address intricate scenarios. We hope our model can support professionals in the arduous clinical diagnosis process.

Meta-review: The paper obtains positive reviews from three knowledgeable reviewers. After rebuttal and discussions, the three reviewers rate the paper as 1 'Weak Accept' and 2 'Borderline Accept‘. Overall, the reviewers are happy about the key idea of coordinating anatomica and textual prompts for better organ and tumor segmentation as well as the experimental evaluation.  

I recommend the acceptance of the paper.
Decision: Accept (poster)