SegVol: Universal and Interactive Volumetric Medical Image Segmentation
Authors: Yuxin Du
Fan BAI
Tiejun Huang
Bo Zhao
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Volumetric Medical Image Segmentation
3D Segmentation Foundation Model
Universal and Interactive 3D Segmentation
Primary area: machine_learning_for_healthcare
TL;DR: We propose a foundation model for universal and interactive volumetric medical image segmentation, trained on the collected 90K unlabeled and 6K labeled data.

Abstract: Precise image segmentation provides clinical study with instructive information. Despite the remarkable progress achieved in medical image segmentation, there is still an absence of a 3D foundation segmentation model that can segment a wide range of anatomical categories with easy user interaction. In this paper, we propose a 3D foundation segmentation model, named SegVol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes, this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. To facilitate efficient and precise inference on volumetric images, we design a zoom-out-zoom-in mechanism. Extensive experiments on 22 anatomical segmentation tasks verify that SegVol outperforms the competitors in 19 tasks, with improvements up to 37.24\% compared to the runner-up methods. We demonstrate the effectiveness and importance of specific designs by ablation study. We expect this foundation model can promote the development of volumetric medical image analysis. The model and code are publicly available at https://github.com/BAAI-DCAI/SegVol.

Seg Vol: Universal and Interactive Volumetric Medical Image Segmentation
Yuxin Du Fan Bai Tiejun Huang?*4, Bo Zhao 12f
School of Artificial Intelligence, Shanghai Jiao Tong University
'BAAT The Chinese University of Hong Kong "Peking University Coorespponingg author: Bo Zhao <bo.zhao witchaaosju..e...c.
Abstract
Precise image segmentation provides clinical study with instructive information. Despite the remarkable progress achieved in medical image segmentation, there is still an absence of a 3D foundation segmentation model that can segment a wide range of anatomical categories with easy user interaction. In this paper, we propose 3D foundation segmentation model, named Seg Vol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes, this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. To facilitate efficient and precise inference on volumetric images, we design experiments on 22 anatomical segmentation tasks verify that SegVol Vol outperforms the competitors in 19 tasks, with improvements up to 37.24% compared to the runner-up methods. We demonstrate the effectiveness and importance of specific designs by ablation study. We expect this foundation model promote the development volumetric medical image analysis. The model and code are publicly available at: http::/ggitub.co//BAAAACCC.C
moom-out-zoom--a in mechanism. Extensive
Introduction
Volumetric medical segmentation, involving extracting 3D regions of interest, such as organs, lesions, and tissues, plays pivotal role in medical image analysis by accurately modeling the 3D structural information of the human body from volumetric medical images such as CT or MRI. The accurate segmentation benefit numerous clinical applications including tumors monitoring
surgical
planning 4 disease diagnosis{3), therapy optimization 6 77, etc.
Compared to 2D medical image segmentation g image segmentation is notably more challenging due the aabor-intensive annotation and resource- consuming computation. Recently, the research of volumetric medical image segmentation has 22 23 However,
9
10
15
16
volumetric
garnered substantial attention, leading to a series of advancements [8] 19 20 21
existing volumetric medical segmentation methods have several key limitations which prevent their application in challenging tasks, e.g., liver tumor or colon cancer segmentation
24 25 26 27), and
real-world tasks, e.g., numan-interaction segmentation 28
29
30
31 32..
Firstly, the publicly available volumetric medical image datasets usually consist of small number of mask annotations from few varying categories. Due to the different label spaces, the traditional task-specific segmentation models trained on one dataset have difficulty in generalizing to others. For
example, the CT-ORG dataset 33 34 24 35) contains the 'lungs' category, while this category is split into two sub-classes and named 'left lung' and 'right lung' in the LUNAI6 dataset (36). Hence, universal segmentation model has to understand the semantics of anatomical categories. Secondly,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Volumetric input
Spatial prompt
Semantic prompt
Liver
Kidney
Spleen
Liver
Pancreas
Kidney
Image
Spattal
Image embeddings
Prompts
Spatial embeddings
Semantic embeddlnng..
Spleen
per pretictiion
Fusion encoder
Pancreas
SegVol
Mask decoder
Mask embeddings
Resizo
Resize
Liver
SegVol
Volumetric Input
prompts
Prediction mask
Back-ff1
window
Rebuidd
Window
Generated prompts
predictions
ROI prodiction
Spatial prompt
prompt
Figure Overview of SegVol model architecture. Seg Vol produces precise segmentation of 3D anatomical structures from volumetric inputs with easy user interactions, including point, bounding box, and text prompts. Zoom-out-zoom-in mechanism: Seg Vol initially produces rough prediction mask with zoom-out inference, then refines it with zoom-in inference on the identified ROI.
traditional segmentation models have inferior performance when segmenting complex structures, such tumors and cysts (37). This is because these models trained insufficient data and also not able to leverage the spatial information through user interaction. Thirdly, previous solutions computationally expensive in the inference process. They typically employ sliding window to infer the whole volumetric input. This strategy is not only time-consuming but also short-sighted, as the sliding window contains only local information. Recently, there have been some works 29 38 39)
that introduce spatial-promph prommt into medical image segmentation, shown in Table However, most of them lack the ability to process the 3D input directly and naturally, and none of them able to understand the semantics of anatomical categories.
In this paper, propose the first foundation model for volumetric medical image segmentation Seg Vol. The proposed model enables universal and interactive 3D segmentation of more than 200 anatomical categories, supporting both spatial and semantic prompts. Seg can also be driven by the combination multi-prompt, like "bounding box+text' or 'point+text' prompts, achieving high- precision segmentation and semantic disambiguation. To enable efficient and precise segmentation of volumetric images, we develop a zoom-ouu noom-out-zoom-in mechanism that enables the model to be efficient and precise. We evaluate the proposed Seg on 22 volumetric medical image segmentation tasks and the results demonstrate our method surpasses other SAM-like interactive segmentation 29] by large margin. Extensive case studies and ablation experiments also carried out to prove the advantages of Vol Vol and the effectiveness of the zoom-out-zoom-in mechanism and multi multi-prompt combination.
methods 28 38 39
We summarize our key contributions as follows:
1. Collect and process 25 public volumetric medical segmentation datasets, encompassing over 200 anatomical categories. The pseudo label is introduced relieve the spurious correlation in the training data.

---

Table 1: The different settings and functions of SAM-like interactive segmentation methods.
Prompt Type
Method
Image Domain
Dimension
Training
Point
Bbox
Text
Inference Input
SAM
Natural
Full-Param
1024x
McdSAM 29
Medical
2D
Decoder
x
X
1024x 1024
SAM-Mcd2D 38
Medical
2D
Adapter
x
1024x 1024
SAM-Med3D
39
Medical
3D
Full-Param
X
x
128 x28x 128
OURS
Medical
3D
Full-Param
V
v
Full Resolution
2. Implement massive 3D pre- pre-training on 96K CT volumes and supervised fine-tuning on the 6k labeled datasets.
Support spatial-pne ppommpt, semantic-promp, and combined-prompp prompt segmentation, achieving high-precision segmentation and semantic disambiguation.
4. Design a zoom-out-zoom: mechanism that significantly reduces the computational cost, meanwhile preserving precise segmentation.
Methodology
2.1 Dataset Construction
One of the main challenges of training universal volumetric medical segmentation model is the absence of large-scale publicly available volumetric medical data, especially CTs with segmentation annotations. Doing utmost, collected 25 open- segmentation CT datasets, including
CHAOS 40 41 42), HaN-Seg 443.
AMOS22 AbdomenCT-1kk (InnennTT-kk45)) [45), KiTS23 (6), KiPA22
48 49 50 5TT BTCV Pancreas-C" 53
3D-IRCADB 55), FLARE22 (6.. (5), 60, 61). SLIVERO7 QUBIQ(63), six MSD datasets (56). LUNA and WORD(64). Their detailed information and availability are shown in the Section A These CTs originate from various medical institutions, captured by different machines with varying parameter settings and scanning regions. To standardize these datasets, we use the mean voxel value of each volume to filter the background and then perform normalization on the foreground voxels.
54
35,,
TotalSegmentator (58) CT-ORG 33
34
24
35),
VerSe19, 59
Volumetric segmentation datasets suffer from the notorious problem of partial labels. Most of these datasets have annotations of only few segmentation targets, e.g., several organs. Therefore, the deep models may learn the spurious correlation between datasets and segmentation targets, and thus produce inferior results during the inference phase. To relieve this problem, introduce the pseudo labels by utilizing the Telzenswalb-Huuttenllddl each CT scan. Pseudo masks can supplement unlabeled categories in dataset, therefore relieving the spurious correlation problem. To restrain the noise and numerous tiny masks in pseudo labels, we employ the following strategies: 1) The pseudo masks are replaced with ground truth masks when applicable. 2) We filter out tiny structures smaller than 1% of the whole volume size. 3) Each mask refined by dilation and erosion operations.
(65) algorithm to generate pseudo masks for
2.2 Model Architecture
Motivated by the recent advance in 2D nature image segmentation, Segment Anything (SAM) 28), design novel model for interactive and universal volumetric medical image segmentation, named, Vol. The model is illustrated in Figure Seg Vol supports three types of prompts for interactive segmentation: "bounding box(bbox) prompt, including the coordinates of two diagonal vertices; 'point' prompt, composed of a set of positive and negative points; and 'text' prompt, such as 'liver' or 'cervical spine C2'. The model consists of four modules: image encoder, text encoder, prompt encoder, and mask decoder.
We employ 3D VIT (Vision Transformer) 66 67
the image encoder, which exhibits remarkable advantages over convolutional models when pre-trained on large-scale datasets. The ViT structure is designed as follows: patch size=(4. 16, 16), layers number= 12, heads number=12, hidden

---

size=768. We first pre-train 3D VIT using SimMIM algorithm[66 669 on the collected 96K CTs, and then conduct further supervised fine-tuning on the 6K CTs with 150K labeled segmentation masks.
One of the main limitations of traditional segmentation models is that the models learn dataset-specific labels encoded as integers which cannot generalized to unseen datasets or tasks, preventing their real-world applications. We enable universal segmentation across datasets by leveraging the text encoder from CLIP model(70) to encode the input text prompt, as CLIP 70] has been trained to align image and text embeddings on web-scale image-text pairs. Given a word or phrase as the text prompt, we complete it using the template 'A computerized tomography of a [text prompt]' [Z]] and then encode it into text embedding. The off- off-the-shee the-shelf text encoder is frozen during training due to the limited text data in CT datasets. Following SAM(28), we obtain the prommpt embedding using positional encoding [72 on point and bbox prompt.
After obtaining the image embedding and prompt emhedding, we input them into the mask decoder and predict the mask. We use self-attention and cross-attention two directions to fuse the image embedding and prompt embedding, and then employ the transposed convolutions and interpolation operations to generate masks. Since text embedding is the key to universal segmentation and it is also challenging to learn the correlation between text and volumetric regions, enhance the text information by introducing a parallel text input branch beside the joint prompt embedding.
2.3 Prompt Generation
SegVol accepts multiple types of prompts, including individual point, bbox, and text prompts, and also their combinations. To make full use of the segmentation training data, we generate kinds of prompts for each datum and construct kinds of prompt-mask data pairs for training.
The point prompt is built from ground truth pseudo masks, consisting of three kinds of points, namely, positive point, negative point, and ignored point. Positive point means that is within the target mask region, while negative points are those outside. Ignored points utilized to ensure a uniform length of the point prompts for input completion. Notably, these ignored points are not considered by the model.
The bbox prompt is generated based on the ground truth or pseudo masks, integrated with random jitter to enhance the model's robustness. When generating the bbox prompt for some pseudo mask, the bbox may also cover other masks due to the irregular 3D shapes. To address this problem, we compute the Intersection over Union (IoU) between the generated bbox and the included pseudo masks. Any mask with an IoU greater than 0.9 will also be integrated and considered as part of the target mask corresponding to this bbox prompt.
The text prompts are constructed based on their category names. As pseudo masks produced by the unsupervised FH algorithm/65) do not have the semantic information, we only use point and bbox prompts for training on masks of pseudo labels.
2.4 Zoom-out-zoom-in Mechanism
SAM-like interaction with large-volume images is laborious for users, especially in the scene where the sliding window has to be used due to the limited view of those 3D models. To provide users with an easy SAM-like interface, we design a coom-out-zoom--a mechanism, which efficient and precise, consisting of moom-out-zoom--e inference and multi-size training. As demonstrated in Figure I the zoom-out process involves resizing a volumetric image, which is input into the model with user prompts to generate a coarse segmentation mask. Then, the Region of Interest (ROI) from the original image is cropped for zoom-in analysis. In the zoom-in process, a sliding window is used to perform precise inference driven by prompts generated from the coarse segmentation mask. After that, the ROI prediction mask will be back-filled to the coarse segmentation mask to finish the final prediction. Besides, multi-size training involves augmenting the input data by resizing CT's for the zoom-out view and cropping them into cubes for the zoom-in view. The zoom-out mechanism realizes the computational cost reduction meanwhile producing precise segmentation of the ROI.

---

2.5 Loss Function
We apply SimMIM algorithm [69] to pre-train the image encoder of SegVol with the masked image modeling loss Lrveiiaiiing Gi:: D1). The loss function is as follows:
Epre-mining (0IE; Di)
|b aMilis
(I)
(2(am)
where Pi: is the parameter of Seg Vol's image encoder. a. DxH*W the input voxel values and predicted values, respectively. M denotes the set of masked voxels, Q(-)) is the number of elements, and D is the pre-training dataset.
combine the Binary Cross-Emin Cross-Entropy (BCE) loss and Dice loss as the supervised fine-tuning loss function Liinetuming ( D2) to train the model with trainable parameters 0 (text encoder frozen). D2 is the supervised fine-tuning dataset and y E R xx are the predicted mask and ground-truth mask, respectively. F(., 0) is the forward function of Seg The loss function is as follows:
LOC (, 0))) (1 y, log(1 F(x, 0))
(2)
2 (y, F(x, 0))
Dice (0, = (z)
(3)
1 1 + F 0
Line tuning (0; D2) = LBCB 0 D2) + Licce (0; D2) The detailed fine-tuning algorithm of Seg Vol is presented in Section B
(4)
3 Experiments
In this section, we conduct extensive experiments on 22 volumetric medical image segmentation tasks to compare Seg with other SAM-like medical image segmentation methods(28) 38 39] 29). Ablation studies are also carried out to prove the effectiveness of the zoom-out-zoome and provide more insights about dataset scale and multi-prompt combination. Detailed case studies are conducted to discuss the disambiguation ability of semantic- promppt and the capability of identifying the segmentation results with spatial- -promp...
mechanism
3.1 Experimental Setup
During the pre-training, we follow SimMIM algorithm(69) to train the 3D ViT encoder of Seg Vol on the collected 96K CTs for 2000 epochs. In the supervised fine-tuning stage, train Seg Vol (with the text encoder frozen) on the labeled 25 volumetric medical image segmentation datasets for 270 epochs with batch size 32 and input size (32, 256, 256), using Adam W o3timizer(73). SimMIM pre-training takes about 20 x GPU hours, while fine-tuning takes about 300 x 8 GPU hours. All the above training process is implemented 8 NVIDIA A A100-SXM4440GB. Three external datasets [44] 74, 755 and 20% testing data preserved from 25 collected datasets are used in the following experiments.
3.2 Compared with SAM-like Interactive Methods
Several efforts have been made to construct a SAM-like interactive medical image segmentation model. However, some of these works, such MedSA.M(9] and SAM-MED2D (38). focus on 2D tasks and cannot process 3D input directly. The other 3D-based methods, such as SAM-MED3D (9), only support small cropped input and do support cemantic-prompt segmentation, which are still far from building a comprehensive foundation model for volumetric medical image analysis.
Competitors and configures. this experiment, MedSAM 29] and SA SAM(bounding box) 28 use bounding box prompts. SAM(5 clicks) SAM-MED2D 38 and SAM-MED3D use point prompts and five-step correction procedure, which means that the point prompt in each step will be given according the previous-step output and ground truth, rather than giving all once. In
5

---

Table Quantitative comparative experiment results for SegVol and other 5 SAM-like interactive segmentation methods settings in terms of the median value of Dice score.
SAM(Point) SAM(Bbox) SAM-MED2D SAM-MAT MED3D MedSAM
Dataset
Category
OURS
28
28
29
Aorta
0.7267
0,4362
0.8704
0.8102
0.3387
0.9273
Bladder
0.4162
0.6281
0.8417
0.4338
0.6799
0.9120
Duodenum
0.1554
0.3192
0.5066
0.3820
0.3066
0.7402
Esophagus
0.2917
0.3541
0,5500
0.5174
0.3610
0.7460
Gallbladder
0.2831
0.6161
0.7999
0.5643
0.6609
0.8763
Adrenal gland(I.)
0.0555
0.4222
0.5068
0.4584
0.3766
0.7295
AMOS22
Left kidney
0.8405
0.8274
0,9325
0.8723
0.7909
0.9489
4
Liver
0.7477
0.5124
0.6904
0.8801
0.6137
0.9641
Pancreas
0.2127
0.3392
0.5656
0.539T
0.3217
0.8295
Postcava
0.2042
0,5251
0,4436
0.6683
0,5211
0.8384
Prostate uterus
0.2344
0.6986
0.7518
0.6231
0.7739
0.8557
Adrenal (R)
0.0452
0.3642
0.1681
0.3708
0,3855
0.6994
Right kidney
0.8459
0.8215
0,9077
0.8632
0.7851
0.9505
Spleen
0.5936
0.6536
0.9267
0.8591
0.7038
0.9589
Stomach
0.4229
0,3883
0,5399
0,4576
0.4378
0,9123
Average
0.4050
0.5271
0.6668
0.6200
0.5371
0.8593
ULS23
DeepLesionn3
0.3686
0.7473
0.3258
0.2386
0.7680
0.7065
BoneLesion
0,4461
0.6671
0.1947
0.4447
0.6896
0.6920
74
PancrcasLesion
0.0675
0.5579
0,5548
0.5526
0.6561
0.7265
Average
0.2941
0.6574
0.3584
0.4120
0.7046
0.7046
Aorta
0.2744
0.3894
0.8077
0.7703
0.3278
0.8439
Seg THOR
Esophagus
0.0348
0.2046
0.3578
0.6394
0.2196
0.7201
Heart
0.6695
0.8876
0.6012
0.8325
0.8924
0.8172
Trachea
0.9147
0.1611
0.8306
0.8485
0.1261
0,8807
Average
0.4734
0.4107
0.6493
0.7727
0.3915
0.8155
Aorta
Left kidney
Right Kidney
Duodenum
Bladder
Pancreas lesion
Right adronaa gland
Prostateiuterus
Postcava
Pancreas
Liver
Bone lesion
Left adrenal gland
Gallbladder
Esophagus
Stomach
Spleen
Desplesion3D
AMOS22
Ours
MedSAM
SegTHOR
SAM-bbox
SAM-MED3D
SAM-point
Aorta
Esophagus
Trachea
Figure Violin plots for quantitative comparison experiment results of Seg Vol and SAM-like The vertical axis represents the Dice score.
interactive methods 28
38
39
29].
this experiment, Seg uses bounding box and text prompt which performs better than other kinds of prompt combinations. Detailed ablation study on prompt combination is demonstrated in Figure 3 (b). In addition, we compare Seg Vol with traditional task-specific segmentation models, e.g.. 3DUX-NET(2)) SwinUNETR(20) and nnU-Net(22) in Section C though the direct comparison is unsuitable due the different settings and objectives.
Testing data. To compare with these SAM-like interactive segmentation models, we evaluate the models on 1,778 cases from the validation set of AMOS22 44), the whole novel annotated set of Uni-

---

Table Ablation experiment on the zoom-out-zoom, in mechanism.
Mechanism
Dice Score Avg.
Time Per Case Avg. &
Resize
0.4509
65 ms
Sliding window
0.6529
3331 ms
Zoom-out-zoome
0.7298
190 ms
Prompt
Dicedh)
Point+loxt
Trachea
600
Text
Bbox
Text-boox
Point
Text
77.17
73.07
Point
Boox+led
Clavcull
Giteey B
List
808
10
CT number
Pointr text
78.29
Text
Stanash
Lung
10
Mask number
Dice Score(8i
Scapuda
79.69
104
60
Fanorcos
83.02
10
100
Easphagae
10a
20
Vencoite
10
RSh
0
2
8
25
Azrunal
Disa Scoee())
Dataset Number
(a)
(b)
Figure 3: (a) The performance of Vol improves the training data scales up. (b) The quantitative experimental results on 19 anatomical segmentation tasks of split 20% test data demonstrate that using the combination of semantic and spatial prompts can achieve better performances.
versal Lesion Segmentation Challenge 23(ULS233 74), and the released labeled set of SegTHOR 75) The validation set of AMOS22 contains 120 cases annotated with 15 major organs. The novel annotated ULS23 dataset composed three subsets, namely, DeepLesion3D, Radboudume Bone, and Radboudume Pancreas. The DeepLesion3D subset contains 200 abdominal lesions, 100 bone lesions, 50 kidney lesions, 50 liver lesions, 100 lung lesions, 100 mediastinal lesions, and 150 assorted lesions cases. There are 744 bone lesion cases in the Radboudumc Bone subset and 124 pancreas lesion cases the Radboudume Pancreas subset. The 40 cases from SegTHOR, which are contoured manually by experienced radiotherapist, focus on the heart, trachea, aorta, and esophagus that surround the tumor and must be preserved from irradiations during radiotherapy.
Quantitative results. The quantitative results of comparative experiments are shown in Table 2 which verify our method is the best in most of the tasks including both lesions and organs, compared
to other SAM-like interactive models 28 38 39 29). Specifically, our method outperforms the second-ranked SAM-MED2D (average Dice score). On the SegTHOR dataset, our method surpasses the runner-up - SAM-MED3D by an average Dice score improvement of 4.28%. The ULS23 dataset, characterized by small patch- like masks, presents a unique challenge. In this scenario, Seg Vol still exhibits good performance, comparable to MedSAM, which excels in using bbox prompts for segmenting small objects. We visualize the Dice score distributions of all methods in all the tasks as violin plots, depicted in Figure More detailed results and visualization are present Section C
the AMOS22 dataset by significant improvement of 19.25%
2
3.3 Ablation Studies
Zoom-out-zoom-in mechanism. One of the key designs of Seg Vol is the zoom-out-zoon mechanism. We compare it with the intuitive resize strategy and the popular sliding window algorithm on the split 20% test data, 48 cases covering 15 major organs with a variety of sizes, belonging to the AMOS22 44 dataset. Two evaluation dimensions, i.e., performance (Dice score) and inference time cost (per case), are compared, as shown in Table The coom-out-zoom-- mechanism achieves the best average Dice score and very competitive inference speed compared to the simple resize strategy. The reason for computational cost reduction that the traditional sliding window method requires scanning the entire 3D CT and processing thousands of windows. In contrast, the proposed zoom-out-7oommin mechanism only requires one global inference of 3D CT and then scanning the ROI with dozens of windows. Detailed experiment results are shown in Section
7

---

Scaling up training data. The success of scaling up training data has been witnessed in multiple
computer vision tasks [28 70]. We conduct an ablation study to investigate the importance of scaling up training images and masks. The split 20% test data of BTCV [52), which includes 13 main organs, is set as an anchor to evaluate the model trained separately on 1, 2, and 8 datasets for 500 epochs, as well as the final model trained on 25 datasets. The detailed results are shown in Figure (a). As a lightweight model, the performance of SegVol is weak when only one dataset is used. However, with the increase of training data, the Dice score increases rapidly, especially in the text prompt setting. The results indicate that our method is scalable and better performance can be achieved if more training data is available.
Multi-prompt combination. As a universal model, our approach achieves precise segmentation for over 200 organs, tissues, and lesions using both spatial and semantic prompts. In Figure B (b), we quantitatively analyze the mutually supportive relationship between semantic-prompt and ppatial-prompt in 19 segmentation tasks of the 20% split test data. On the one hand, spatial-promp allows the model locate the specific part in the 3D space. According to Figure (b), the average Dice score of the 'bbox+text' prompt is boosted by 5.85% compared to the 'text' prompt on average. On the other hand, semantic-promp clarifies the reference the anatomical structure, eliminating the ambiguity of spatial-promph and the plausible masks of multiple categories. This is reflected in Figure 3 (b) as the average Dice score of 'point+text' prompts is 4.62% higher than using 'point' prompts alone. Spatial and semantic prompts mutually support each other, ultimately endowing the model with powerful segmentation capabilities.
3.4 Case Studies
Disambiguation via cemantic-prompl It a notorious problem in interactive segmentation that one spatial-prompt may correspond to multiple plausible outputs (28). As illustrated in the images on the top left in Figure three of them correspond to three anatomical concepts, namely, kidney tumor, left kidney, and the whole kidneys, while they are all plausible to the same point prompt. Similarly, in the bottom left three images, the bounding box selects the region of the liver. However, liver tumors, hepatic vessels, and the liver itself are also plausible target structures. In these cases, SAM chooses to return multiple masks to match different levels of plausible results. Unlike SAM's solution, we use remantic-poomp to clarify the targets. As shown in Figure 4 the captions below the images are the text prompts, and the masks in the images are the predictions of Seg Vol, which show that monantic-prompl can effectively disambiguate the spatial-prompt.
Identifying the spatial-prompt segmentation. Furthermore, we study the capability of Seg Vol to identify the semantic category of the spatial-prompt results. Figure Freveells that Seg Vol can give accurate semantic categories based the spatial- -poompt results. In the top left image in Figure 5 the spatial-prompt prompt on the liver results in a 0.997 prediction score for the liver. The top right image in the sub-figure shows if the spatial-promp is the point on the liver tumor, Sel Vol will output 0.619 prediction score for the tumor category and a 0.339 prediction score for the liver based on the spatial relationship of liver tumor and liver. We implement this identification experiment by decoding the semantic prompts from category set. The softmax function is applied to the decoding results to get the prediction probabilities of different categories. The probabilities on the initial predicted mask, driven by the spatial-promph are used to calculate the final classification result.
4 Discussion
Scalability. The scaling law of foundation models has been verified in multiple CV and NLP tasks. Since Seg uses transformer- architecture and self- supervieee pre-training algorithm, it has strong data and architecture scalability. this work, we achieve the success of scaling law in 3D medical segmentation by the design of universal prompts and pseudo masks for joint learning on datasets with inconsistent annotations. The ablation study of scaling up training data shows that 1) the performance improves significantly with more training data in the 3D segmentation task, 2) Seg Vol has not yet reached its ceiling if more training data provided. We believe the performance of Seg Vol can be continuously improved when more data and computational resources used.
8

---

Kidney Tumor
Kidney
Kidneys
Lung Tumor
Right Lung
Lungs
Liver Tumor
Hopatic Vessels
Liver
Liver
Aorta
Stomach
Figure 4: The four cases demonstrate that comantic-prompyy can clarify the ambiguity of spatial- prompt and avoid multi-plausibi outputs. Each image shows the segmentation result of Seg using the spatial-prompt, i.e. point or bounding box, and semantic-promp: i.c. the caption below the image.
Liver
0.997
0.619
Tumor
0.003
Liver
0.339
Aorta
0
Heart 0.041
Lung
Heart
0
Right Lung
0.001
Right Kidney
Trachea
Left Lung
0
Gallbladder
Pancreas
0
Right Kdney
0.6
Pancreas
0.857
Pancreas
0.577
Stamach
0.072
Tumor
0.423
Stomaih
Liver 0.002
Spicen
Trachea
Right Lung
Right Kidney
Right Kidney
Aorts
Liver
Right Lung
0.987
Trachea
0.971
Liver 0.009
Left Lung
0.017
0.008
Right Lung
Lung
Heart 0.001
Heart
Tumor
Tumor
0
Stemach
Stomach
0
Splcen
Right Kidney
Right Kidney
Figure 5: We identify the semantic categories of the spatial-promn segmentation results. Each image shows the ppatial-prompp and the mask prediction. The bar charts rank the top 8 semantic categories with the highest classification probabilities. The results show that SogVol is capable of identifying the anatomical category of the segmentation mask using spatial prompts.
Generalizability to unseen modality. Although we develop Seg Vol on Computed Tomography (CT) data due to its advantages of easy acquisition, wide usage, and high resolution, we find that SegVo Vol can generalize other medical image modality, like MRI. Namely, the Seg Vol model trained only on CT data can be used to segment MRI with semantic and spatial prompts. This emerging ability demonstrates that our foundation model understands the anatomical structure of human body, We provide detailed experiments and analysis of this generalizability in Section The impressive generalizability makes Seg Vol versatile tool in medical image analysis. We leave the joint training of Seg Vol on multi-modality data as the future work.
Limitations. Although SegVol shows remarkable semantic-promip segmentation performance, there still remains gap between it and the referring volumetric segmentation. A promising solution is construct the referring segmentation data with diverse semantic and spatial prompts, and then train Vol on it. We leave the future work. More discussions can be found in Section

---

Broader impact. We contribute foundation model for universal and interactive volumetric medical image segmentation, which can benefit numerous clinical study and applications. As foundational research work, we do not see any obvious negative societal impact of the proposed method and model.
5 Conclusion
In this paper, propose SegVol, a universal and interactive volumetric medical image segmenta- tion model, supporting both spatial-promp and semantic-prompt segmentation of more than 200 anatomical categories. We construct large-scale dataset, which consists of 90K unlabeled CTs and 25 open-source medical datasets, to train the foundation model. We design the zoom-out-zoom-1 mechanism to facilitate efficient and precise inference in the region of interest. Extensive experiments on 22 segmentation tasks demonstrate the outstanding performance of our method. Detailed ablation studies are also carried out to prove the effectiveness of the zoom-out-zoom in mechanism, dataset scale, and multi-prompt combination strategy. As foundation model, we helieve that Seg Vol will advance the volumetric medical segmentation and benefit numerous downstream tasks.
Acknowledgements.
This work is funded by NSFC-62306046.
References
[1] Sidra Sajid, Saddam Hussain, and Amna Sarwar. Brain tumor detection and segmentation in mr images using deep learning. Arabian Journal for Science and Engineering, 44, 2019.
[2] Stefano Trebeschi, Zuhir Bodalal, Thierry N Boellaard, Teresa M Tareco Bucho, Silvia G Drago, Ieva Kurilova, Adriana M Vaina,, Andrea Delli Pizzi, Mirte Muller, Karlijn Hummelink, et al. Prognostic value of deep learning-mediatte treatment monitoring in lung cancer patients receiving immunotherapy. Frontiers in Oncology, 1 11:609054 2021.
[3] Jordi Minnema, Anne Ernst, Maureen van Eijnatten, Ruben Pauwels, Tymour Forouzanfar, Kees Joost Batenburg, and Jan Wolff. A review on the application of deep learning for et reconstruction, bone segmentation and surgical planning in oral and maxillofacial surgery. Dentomaxiilofacial Radiology, (1)7):002144)) 2022.
[4] Vincenzo Ferrari, Marina Carbone, Carla Cappelli, Luigi Boni, Franca Melfi, Mauro Ferrari, Franco Mosca, and Andrea Pictrabissa. Value of multidetector computed tomography image segmentation for preoperative planning in general surgery. Surgical endoscopy, 26, 2012.
[5] Chen Chen, Chen Qin, Huaqi Qiu, Giacomo Tarroni, Jinming Duan, Wenjia Bai, and Daniel Rueckert. Deep learning for cardiac image segmentation: review. Frontiers in Cardiovascular Medicine, 7:25, 2020.
[6] Gihan Samarasinghe, Michael Jameson, Shalini Vinod, Matthew Field, Jason Dowling, Arcot Sowmya, and Lois Holloway. Deep learning for segmentation in radiation therapy planning: a review. Journal of Medical Imaging and Radiation Oncology, 65(5):578-595 2021.
[7] Habib Zaidi and Issam El Naqa. Pet-guided delineation of radiation therapy treatment volumes: survey of image segmentation techniques. Juropeanjournal of nuclear medicine and molecular imaging, 37:2165-2188 2010.
[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence. 40(4):834-848, 2017.
[9] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang, Unet++: nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop. DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, pages 3-11. Springer, 2018
[10] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: review of theory and applications. IEEE Access, 9:82031-82057, 2021.
10

---

[11] Xiao-Xia Yin, Le Sun, Yuhan Fu, Ruiliang Lu, Yanchun Zhang, et al. U-net-based medical image segmentation. Journal of Healthcare Engineering, 2022, 2022.
[12] Jiawei Zhang. Yuzhen Jin, Jilan Xu, Xiaowei Xu, and Yanchun Zhang. Mdu-net: Multi-scale densely connected u-net for biomedical image segmentation. arXiv preprint arXiv:1812.00352, 2018.
[13] Mehreen Mubashar, Hazrat Ali, Christer Grönlund, and Shoaib Azmat. R2u++: multiscale recurrent residual u-net with dense skip connections for medical image segmentation. Neural Computing and Applications, 34(20):17233-773)
2022.
[14] Debesh Jha, Michael A Riegler, Dag Johansen, Päl Halvorsen, and Håvard D Johansen. Doubleu- net: A deep convolutional neural network for medical image segmentation. In 2020 IEEE 33rd International symposium on computer-based medical systems (CBMS). IEEE, 2020.
[15] Ziang Zhang, Chengdong Wu, Sonya Coleman, and Dermot Kerr. Dense-inception u-net for medical image segmentation. Computer methods and programs in biomedicine, 192:
2020.
[16] Zaiwang Gu, Jun Cheng, Huazhu Fu, Kang Zhou, Huaying Hao, Yitian Zhao, Tianyang Zhang, Shenghua Gao, and Jiang Liu. Ce-net: Context encoder network for 2d medical image segmentation. IEEE transactions medical imaging, 38( 28(10) 2019.
[17] Zaiquan Yang, Yuhao Liu, Jiaying Lin, Gerhard Hancke, and Rynson W. H. Lau. Boosting weakly-supervised referring image segmentation via progressive comprehension, 2024.
[18] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574-584, 2022.
[19] Hong-Yu Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu. nnformer: Interleaved transformer for volumetric segmentation, 2022.
[20] Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images, 2022.
[21] Yucheng Tang, Dong Yang. Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10730-207400 2022.
[22] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep sarning-based biomedical image segmentation. Nature methods, 2822 2021.
[23] Ho Hin Lee, Shunxing Bao. Yuankai Huo, and Bennett A. Landman. 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation, 2023.
[24] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 1026800 2023.
[25] Patrick Ferdinand Christ, Florian Ettlinger, Felix Grün, Mohamed Ezzeldin A Elshaera, Jana Lipkova, Sebastian Schlecht, Freba Ahmaddy, Sunil Tatavarty, Mare Bickel, Patrick Bilic, et al. Automatic liver and tumor segmentation of ct and mri volumes using cascaded fully convolutional neural networks. arXiv preprint arXiv:170..05977 2017.
[26] Ishak Pacal, Dervis Karaboga, Alper Basturk, Bahriye Akay, and Ufuk Nalbantoglu. A com- prehensive review of deep learning in colon cancer. Computers in Biology and Medicine, 26: 104003, 2020.
[27] A Ben Hamida, Maxime Devanne, Jonathan Weber, Caroline Truntzer, Valentin Derangère, François Ghiringhelli, Germain Forestier, and Cédric Wemmert. Deep learning for colon cancer histopathological images analysis. Computers in Biology and Medicine, 136: 104730, 2021.
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint 23X1v:2304.026443 2023.
11

---

[29] Jun Ma, Yuting He. Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images, 2023.
[30]
Hiba Ramadan, Chaymae Lachqar, and Hamid Tairi. A survey of recent interactive image segmentation methods. Computational visual media, 6:355-384, 1055-384,20 2020.
[31] Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han. Linguistic structure guided context modeling for referring image segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part X 16, pages 59-75. Springer, 2020.
[32] Fan Bai, Yuxin Du, Tiejun Huang, Max Q H. Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models, 2024.
[33] Blaine Rister, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org: Ct volumes with multiple organ segmentations [dataset]. The Cancer Imaging Archive, 2019.
[34] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel I. Rubin, Ct organ segmentation using gpu data augmentation, unsupervised labels and iou loss. arXiv preprint 1r /226, 2018
[35] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia):: maintaining and operating a public information repository. Journal of digital imaging,
26:1045-1057, 2013.
[36] Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge. Medical image analysis, 42:1-13, 2017.
[37] Huiyan Jiang, Zhaoshuo Diao, and Yu-Dong Yao. Deep learning techniques for tumor segmen- tation: review. The Journal of Supercomputing, 78(2):1807-188 2022.
[38] Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sam-med2d. arXiv preprint 2338.: 16184, 2023.
[39] Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu, Shaoting Zhang, Junjun He, and Yu Qiao. Sam-med3d, 2023.
[40] A. Emre Kavur, N. Sinem Gezer, Mustafa Bans,, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, et al. Chaos challenge- combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis, 69: 101950, April 2021.
[41] Ali Emre Kavur, M. Alper Selver, Oğuz Dicle, Mustafa Barış, and N. Sinem Gezer. Chaos combined (ct-mr) healthy abdominal organ segmentation challenge data. April 2019.
[42] A. Emre Kavur, Naciye Sinem Gezer, Mustafa Bariş, Yusuf Şahin, Savas Özkan, Bora Baydar, Ulaş Yüksel, Çağlar Kılıkçıer, Şahin Olut, Gözde Bozdağı Akar, Gözde Ünal, Oguz Dicle, and M. Alper Selver. Comparison of semi-automatic and deep learning based automatic methods for liver segmentation in living liver transplant donors. Diagnostic and Interventional Radiology.
26:11 1-21, January 2020.
[43] Gašper Podobnik, Primož Strojan, Primož Peterlin, Bulat Ibragimov, and Tomaž. Vrtovec. Han-seg: The head and neck organ-at-riss et and mr segmentation dataset. Medical physics. 10(3):1917-1227. 2023.
[44] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma. Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprini arXiv:2206. 20t.:22206..0033.. 2022.
[45] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, Shucheng Cao, Qi Zhang. Shangqing Liu, Yunpeng Wang, Yuhui Li, Jian He, and Xiaoping Yang. Abdomenct- 1k: Is abdominal organ segmentation solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 2022.
12

---

[46] Nicholas Heller, Fabian Isensee. Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang, Alex Golts, et al. The kits21 challenge: Automatic segmentation of kidneys. renal tumors, and renal cysts in corticomedullary-
ct, 2023.
[47] Yuting He, Guanyu Yang, Jian Yang, Rongjun Ge, Youyong Kong, Xiaomei Zhu, Shaobo Zhang, Pengfei Shao, Huazhong Shu, Jean-Louis Dillenseger, et al. Meta grayscale adaptive network for 3d integrated renal structures segmentation. Medical image amalysis, 7 71:102055, 2021.
[48] Yuting He. Guanyu Yang, Jian Yang, Yang Chen, Youyong Kong, Jiasong Wu, Lijun Tang, Xiaomei Zhu, Jean-Louis Dillenseger, Pengfei Shao, et al. Dense biased networks with deep priori anatomy and hard region adaptation: Semi-supervised learning for fine renal artery segmentation. Medical image analysis, 63:101722, 2020.
[49] Pengfei Shao, Chao Qin, Changjun Yin, Xiaoxin Meng, Xiaobing Ju, Jie Li, Qiang Lv, Wei Zhang, and Zhengquan Xu. Laparoscopie partial nephrectomy with segmental renal artery clamping:
technique and clinical outcomes. European wrology, 59(5):449 2011.
[50] Pengfei Shao, Lijun Tang, Pu Li, Yi Xu, Chao Qin, Qiang Cao, Xiaobing Ju. Xiaoxin Meng, Qiang Lv, Jie Li, et al. Precise segmental renal artery clamping under the guidance of dual- source computed tomography angiography during laparoscopic partial nephrectomy. European 2012.
urology, (2(6))
[51] Nicholas Heller, Fabian Isensee, Klaus H Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, et al. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced et imaging: Results of the kits19 challenge. Medical Image Analysis, page 101821, 2020.
[52] Bennett Landman, Zhoubing Xu, Igelsias, Martin Styner, T Langerak, and Arno Klein. Miccai multi-atlas labeling beyond the cranial vault-workshop and challenge. In Proc. MICCAI Aulti-Atlas Labeling Beyond Cranial Saulr-Worksher Challenge, volume 5, page 12, 2015.
[53] Holger R Roth, Amal Farag, E Turkhey, Le Lu, Jiamin Liu, and Ronald M Summers. Data from pancreas-ct. the cancer imaging archive. IEEE Transactions on Image Processing, 2016.
[54] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pan- creas segmentation. In Medical Image Computing and Computer- Assissed Intervention-MIC. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 556-564. Springer, 2015.
[55] Luc Soler, Alexandre Hostettler, Vincent Agnus, Arnaud Charnoz, Jean Saan-Batistte Fasquel. Johan Moreau, Anne- Blandine Osswald, Mourad Bouhadjar, and Jacques Marescaux. 3d image reconstruction for comparison of algorithm database. URL: https://www. ircad. frrresearch/data- hattlirrreeemeetttttttd-----eele
2010.
[56] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneide Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint /902 09063, 2019.
[57] Jun Ma. Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge. Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al. Abdomenct-Ik: Is abdominal organ segmentation solved problem? IEEE Transactions on Patlern Analysis and Machine Intelligence, 44(10):6695-671 2021.
[58] J Wasserthal, M Meyer, HC Breit, J Cyriac, S Yang, and M Segeroth. Totalsegmentator: Robust segmentation of 104 anatomical structures in et images 2022. arXiv, 2022.
[59] Anjany Sekuboyina, Malek E Husseini, Amirhossein Bayat, Maximilian Löffler, Hans Liebl, Hongwei Li, Giles Tetteh, Jan Kukačka, Christian Payer, Darko Štern, et al. Verse: a vertebrae labelling and segmentation benchmark for multi-detector et images. Medical image analysis, 73: 73:102166, 2021.
[60] Maximilian T Löffler, Anjany Sekuboyina, Alina Jacob, Anna-Lena Grau, Andreas Scharr, Malek El Husseini, Mareike Kallweit, Claus Zimmer, Thomas Baum, and Jan S Kirschke. vertebral segmentation dataset with fracture grading. Radiology: Artificial Intelligence, 2(4):e 190138, 2020.
13

---

[61] Hans Liebl, David Schinz, Anjany Sekuboyina, Luca Malagutti, Maximilian T Löffler, Amirhos- sein Bayat, Malek El Husseimi, Giles Tetteh, Katharina Grau, Eva Niederreiter, et A computed tomography vertebral segmentation dataset with anatomical variations and multi-vendor scanner data. Scientific data, 8(1):284, 2021.
[62] Tobias Heimann, Bram Van Ginneken, Martin A Styner, Yulia Arzhaeva, Volker Aurich, Christian Bauer, Andreas Beck, Christoph Becker, Reinhard Beichel, György Bekes, et al. Com- parison and evaluation of methods for liver segmentation from datasets. IEEE transactions on medical imaging, 28(8):1251-1265 2009.
[63]
Quantification of uncertainties in biomedical image quantification challenge 2021. https: org/ Accessed: 18 Aug 2023.
/ g/gublg2l...aandcchhaa....
[64] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song. Xiaofan Zhang, Kang Li, Dimitris N. Metaxas, Guotai Wang, and Shaoting Zhang. WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image. Medical Image Analysis, 82:102642, 2022.
[65] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph- based image segmentation. International journal of computer vision, 59:167-181, 2004.
[66] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[67] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmenta-
tion, 2021.
[68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[69] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling, 2022.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 PMLR, 2021.
[71] Jie Liu, Yixiao Zhang. Jic-Neng Chen, Junfci Xiao, Yongyi Lu, Bennett A. Landman, Yixuan Yuan. Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection, 2023.
[72] Matthew Taneik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020.
[73] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[74] Max de Grauw. Universal lesion segmentation challenge 23. https://ub3..gramddcallllleeeeee grand-challengg.orgg
[75] Z. Lambert, C. Petitjean, B. Dubray, and S. Ruan. Segthor: Segmentation of thoracic organs at
risk in images, 2019.
(76) brgfx. Image by brgfx on freepik. http://www..iee.noooooe.eeeeeons human-body 275.9440..htm.
14

---

A Dataset Details and Availability
In this work, we collect 25 open-source datasets for supervised fine-tuning Seg Vol and some external open-source datasets specifically for comparative experiments. The detailed information on anatomi- cal categories and dataset scales of these open-source datasets is shown in Table The availability of these datasets is demonstrated in Table 5 Additionally, avoid privacy concerns, collect 90K unlabeled CTs from publicly accessible professional medical websites; hutpss//raddiopuediaorr//
The collected segmentation datasets include major regions of the human body, i.e., the head, neck, thorax, abdomen, and pelvis, comprising over 200 categories of organs and tissues, and 28 lesion tasks from different benchmarks. The detailed categories information shown in Figure Figure/oland some representative samples are shown Figure
Table 4: Information of datasets involved in supervised the-ttning and experiments.
Category Number
Trainset Volumes
Dataset
Anatomical Targets
3D-IRCADB 55
Liver and liver tumor
47
20
AbdomenCT-1k445
Liver, kidney, spleen, pancreas
1000
AMOS22
Abdominal organs
15
240
BTCV 22
Abdominal organs
13
30
CHAOS 40
42
Abdominal organs
20
CT-ORG 33
34
24
35
Brain, lung, bones, liver, kidney, and bladder Thoracic and abdominal organs
140
FLARE22 56 57
13
50
HaN-Sog 43
Organs of the head and neck Kidney, renal tumor, artery, and vein Kidney and kidney tumor Kidney, kidney tumor, and kidney cyst Left lung, right lung, and trachea Colon tumor Hepatic vessel and liver tumor Liver and liver tumor Lung tumor
30
42
KiPA22 47
49
50
4
70
KiTS19 [1
2
210
KiTS23 46
3
489
LUNA16[36
3
888
MSD-Colon 56
126
MSD-Ieepatic Vessel 56
2
303
MSD-Liver 56
2
131
MSD-lung 56
63
MSD-pancrcas 56
Pancrcas and pancreas tumor
2
281
MSD-spleen 56
Spleen
41
Pancreas-CT 53 54 35
Pancreas
82
QUBIQ[63
Kidney, pancreas, and pancreas lesion
3
82
SegTHOR 75
Heart, trachea, aorta, and esophagus
40
SLIVERO7
Liver
20
TotalSegmentator (8
Organs the whole body
104
1203
ULS23(novel annotated set) 74
Various lesions
1618
VerSe19
Vertebrae
80
VerSe20
vertebrae
61
WORD
Thoracic and abdominal organs
B Training Algorithm
Due to the complexity of the training steps, which include decoder reuse, the combination of different datasets, and cooperative training of ground-truth and pseudo labels, we abstract the core training code as Algorithm and Figure to clarify the training process of SegVol. As shown in Figure 8 each case (training sample) consists an Image æ, a Ground Truth(GT) Mask Set Y. and a Pseudo Mask Set Z. The training loss of each sample consists of the ground-truth loss and the pseudo loss. The ground-truth loss is computed by inputting the image, the ground-truth mask (label), and sampled prompt into the model, while the pseudo loss is computed by inputting the image, the pscudo
15

---

Cocteajl_
Ducdenum
tyv
FanAl AR PRI
Optedhaamm Optcnrdiiti
Colon
Cakin concrr
Lacrived
Gind lacrrallff)
Kenny
Karsa lonor 3, Rarai weat,
Paroti gland
Parct63//0
Apria
Avji
Fece
PaW
Lierr
Lver Hapalic
Uven cyliiyale
Head and Nock
Subglotic
HyAb404, Chogparsnggnn Eagraag
Pancwa
Panoww, Pancrettcieddng
Orll
Can
Rone masdite, Racaimaaees
Storach
Skamech
Bubmandlhsseer
Died subrerelLPP
Excphague
Styrll
Lumber
Lumber spna
Spiral card
Spiraicard
Carerid artary
Camid))
Interirr vena came
Intvior vereioee
ppra|Ch--C))
Veriebraa
Samacow T1-T12
Body part
bub-rogion
Anelorvicll structurv
Rectum
Reckar
Lon] lnexe kineler portrrgggtt Lung
Lung
Larg apper
Porial vois
Pirid wan
Body part
Sub-region
Aratomica atruction
Heart
wi't witrdejee rigi)
Ohaas mate uadult rights ightt
Purreonary
Pulrerery
Chraso memenndiffuut
Thorax
Aorts
Hamerua
Adtt
Humercalatt egh0
Pates
Famur
Hoad of tomuttattngggg
anceyteryin right
lbac
Tereeds setoot
Thoradi
Disdder
Hib
Ri 1490--112 Pio 10107-111
Sacmer
Saeurr
offedions
Figure 6: Overview of the collected datasets for supervised fine-tuning. The joint dataset comprises 47 important regions, with each region containing one or multiple significant anatomical structures within that spatial area. Image the human body by brgfx on Freepik 76].
Figure 7: The joint dataset encompasses various anatomical structures in major regions of the human body. Several volume examples are demonstrated 2D slices and 3D shapes the images respectively.
16

---

Table Availability of datasets involved in supervised fine-tuning and experiments.
Dataset
Link
3D-IRCADB 55
AhdomenCTT11k
AMOS22 [4]
http:///aama22222gmmdddlllllnnrg
BTCV 52
CHAOS
https//cchaassgrraacccclllleeggg
https///wwii.aaaccrrrnnnggngngge
CT-ORG 33
35
FLARE22 56
57
https///fare22.gaammcccllllggoep
HaN-Seg 45
KiPA22 47
48
50
https///kpa222garabcccchllegggg
KiTS
http://Attsssgggrrmdccchhlllnggr
K4TS23[46]
https///kiiscchallleeeeorkkkkk/k
LUNA 16
http://tnnaatgggaaadcllllleeeeer
MSD-Colon (5
http://meedicaaldecattlloccc
MSD-Hepatic Vessel 56
Vessel
http://meedicaadeecatto.occmm
MSD-Liverr56
http://meedicaalecaatloo..cc
MSD-lung [6
http://meedicaldeccttloo.ccc
MSD-pancreas 56
http://mediiaalecattloo..cmmm
MSD-spleen 56
http://meeticaaleccatllo.cccm
Pancreas-CT 53 54
35
QUBIQ
SegTHOR
SLIVERO7 62
TotalSegmentator 58
ULS23 74
https://ol23.ggaamdcccclllleeg..
VerSe 60 61
https://osf..o//giiwww/
VerSe20 60 61
https://osffio/198tz/
WORD 64
Table 6: Complexity comparison of popular methods.
Total Parameters
Avg.
Avg.
Avg. Dice Score
Method
MACs Per Casel
Time Per Case(s)
SAM
94M
1.3e+13
2.1764
0,5271
MedSAM
94M
1.3c+13
2.1886
0.5371
SAM-MED2D
38
271M
2.3e+12
3.5547
0.6668
SAM-MED3D 39
101M
1.0e+11
0.1768
0.6200
Seg Vol
181M
6.7e+11
0.3283
0.8593
label, and the pre-designed prompt into the model. Finally, the model is optimized by minimizing the weighted sum of the two losses.
Besides, we add reinforcement branch for semantic- prompt the mask decoder. We further compute similarity matrix between the up-scaled embedding from the transposed convolution output and the text embedding. The element-wise multiplication of the similarity matrix with the mask prediction is applied before interpolation, after which the model outputs the masks.
17

---

Algorithm Seg Vol training loop
Input: Seg Vol model, training image a, ground truth mask set Y {ys Mi pseudo mask set
Zr {zi m
Output: Seg Vol model parameters
1: n = # Number of combinations of prompt types: text, point, and bbox.
2: 0.1 # Pseudo loss weight.
3: Loop for each category of this case.
4: for to do
5:
fing modee..maaeeEncoder(r)))
6:
pfapatial ptsemamtic, grompt_generatie
7:
lat
l= -=
8:
LLopp for possible prompt combination types of ground truth mask.
9:
for < 1 to n. do
10:
# Choose prompt combination type.
11:
pt Apaiial semuniic * PromptStrategy (pt spatial, ptacmantic)
12:
fuxl + Sonoee.TexEEncoer(((p) semantic)
fpompt model. Pranee.epomm spatial fixi)
predg model.Decoder( img. prompts Frext)
15:
let < 1et DiceLoss(pr y. BCELoss(predg, y:)
end for
17:
(pseudo $ 0
Loop for several pseudo masks.
19:
for < 1 to n. do
20:
Random select a pseudo mask this case for training.
21:
2p < =RandomSelect(zn mm)
plapatial prompttgenerate! z,p
23:
f fprompt mmo.ee.PromppEnccodrrtt...o
24:
pr predpacdo + Decodel.ecoee((fffe. (figg pprompt)
25:
pscudo Ipccudo DiceLoss(preddeeg 2p) + BCELoss 2p)
26: end for
27: I
Ipseudo (mdate I)
28:
29:
end for
30:
return model
C Additional Experimental Analysis
Comparative experiments to compare with task-specific segmentation models. Task-specific segmentation models mainly fall into architectures. CNN-based models and Transformer-basee models. We conduct comparative experiments with representative CNN-based models i.c. 3DUX- Net 23 and nnU-Net [2]]. and representative Transformer-baas based models i.e. SwinUNETR 20) We conduct additional comparative experiments on the split 20% test set of the datasets. 10 segmentation tasks are selected from BTCV[52] and MSD-spleen (56) datasets, which focus on organ segmentation, and from MSD-lung. MSD-colon, and MSD-liver datasets, which focus on lesion segmentation. We train task-specific segmentation models on each dataset individually for each method.
The quantitative experimental results are summarized in Figure Generally speaking, Seg Vol, jointly trained on 25 datasets, outperforms traditional task-specific segmentation models trained on a single dataset. Compared to these strong baselines, Seg exhibits a narrower distribution of Dice scores across the eight tasks, indicating its robustness and good generalization ability. This mainly owes to the massive knowledge learned from diverse samples of the same categories but different datasets. Seg Vol depicts excellent performance on lesion tasks which are more challenging in semantic understanding and spatial locating. We present detailed comparison to nnU-Ne: on lesion tasks. As shown Table 7 the average Dice score of Seg Vol 14.76% higher than that of nnU-Net for lesion tasks. We visualize the prediction results of the two methods in Figure TO which intuitively show that Seg Vol performs more precise segmentation of the tumors than nnU The detailed scores and visualization results are presented in Table 9 and Figure
and 13
18

---

Training Algorithm
Image
GT Mask y
GT Mask Set Y
Loop
GT Loss
Prompt Types
Case
Image x
Loop
Pseudo Mask Set
Pseudo Loss
GT Mask Set Y
Pseudo Mask Set
Fixed Prompt
The demonstration of the training algorithm. Specifically, each case (training sample) consists of an Image x, a Ground Truth(GT) Mask Set Y, and a Pseudo Mask Set Z. The training loss of each sample consists of the ground-truth loss and the pseudo loss. The ground-truth loss computed by inputting the image, the ground-truth mask (label), and the sampled prompt into the model, while the pseudo loss is computed by inputting the image, the pseudo label, and the fixed prompt into the model. Finally, the model is updated by minimizing the weighted sum of the two losses.
Figure 8:
Ours
nnU-Net
3DUX-NET
SwinUNETR
0.6
0.4
066
Liver
Stomach
Spleen
Pancreas
Esophagus
I
0.
9.85
84
0.6
0.50
0.75
Gallbladder
Left kidney
Liver tumor
Lung tumor
Colon cancer
Figure 9: Violin plots comparing experiment results SolVol Vol and task-specific methods. The vertical axis is the Dice score.
We analyze that there are mainly three factors that make SegVol Vol more powerful than traditional task- specific models: 1) Massive generative pre-training on unlabeled data endows Seg with complete understanding of the volumetric structures and the discriminative feature representations, which is much superior learning from small number of samples. 2) Learning from joint datasets with semantic-prome prompt makes SegVol generalize better to unseen data and categories. For instance, SegVal can learn from both the 'left kidney' and 'kidney' categories based on their semantic correlation. while traditional task-specific models treat the two categories independently. 3) SegVol Vol can be prompted with (spatial) points/bboxes, which provide precise spatial reference, and (semantic) texts, which disambiguate the overlap of multiple categories in the same space. In contrast, traditional methods are able to understand semantics. This ability enables Seg Vol to perform better than traditional methods in challenging tasks, e.g., segmenting lesions.
19

---

Table 7: The comparison of the average Dice score of SegVo, and nnU-Net segmentation tasks.
across lesion
Method
Lung Tumor
Colon Cancer
Liver Tumor
nnU-Net 22
0.5963 0.7122
0.3769 0.6965
0,3606 0.7825
OURS
Ground Truth
nnU-Net
Ours
Ground Truth
nnU-Net
Ours
Liver turorr
Colon tumor
Lung tumor
Figure 10: Visualization results of Seg Vol and nnU-Net across lesion segmentation tasks.
Supplement results for comparative experiments on SAM-like interactive segmentation methods. In this work, compare Seg Vol with 5 SAM-like interactive segmentation methods AMOS22 (44, (742, and Seg SeOTHORRp 75) datasets. The detailed records of Dice score are demonstrated in Table TO, The visualization results shown in Figure [4] Figure [3] Figure [6] In this experiment, Seg Vol is driven by 'bbox+text" prompt. We demonstrate the consistency results among different prompt settings of Seg Vol in Figure 17 which also conducted AMOS22 (44) and ULS23
Relatively poor text prompt results in ULS23 are due to the unclear category of the dataset.
We also compare the Total Parameters, the average multilyy-Accumulats(W/M. Time required to process a case of the different SAM-like methods, as shown in Tabld6 The comparison indicates that method takes less computational cost while achieving much better performance. Note that when calculating MACs Per Case and Time Per Case, the slice-by-slic calculation of the 2D method and the scanning process of the 3D method are accumulated respectively. SAM-MED3D only processes volume with a size of 128 x 128 x 128. The experiments are implemented the validation of AMOS22 44), and the setting the same that in Sec. 3.2
and the average
Supplement results for ablation studies on coom-out-zoom-ie mechanism. We conduct ablation study coom-out-zoom-i mechanism the split 20% test data of AMOS22 dataset. As shown in Table [] the zoom-out-zoom-in mechanism achieves higher Dice scores compared to the
Table 8: Few-shot fine-tuning experiment on FLARE22 56 57 represents the model fine-tuned on all datasets.
and MSD-spleen 56)
Seg Vol#
Avg. Dice Score
100 epochs
200 epochs
300 epochs
400 epochs
500 epochs
Seg
FLARR22 56 57
0.0463
0.4028
0.4926 0.9433
0.5617 0.9454
0.5567
0.8822
MSD-spleen 56
0.7566
0.7866
0.9471
0.9597
20

---

Ground Truth
3DUX-NET
SwinUNETR
nnU-Net
SegVol
Aorta
Left Kidney
Figure 11: Visualized aorta and left kidney prediction results of 3DUX-NET(23) SwinUNETR (20). (22 and Seg Vol on 4 cases from the split test set. For the integrality of aorta and left kidney structure modeling, Vol significantly outperforms 3DUX-NET and SwinUNETR and is comparable to nne-Net.
21

---

Ground Truth
3DUX-NET
SwinUNETR
nnU-Net
SegVol
Liver
Pancreas
Figure 12: Visualized liver and pancreas prediction results of 3DUX-NET SwinUNETR 20]. nnU-Net[22] and SegVol Vol on 4 cases from the split test set. For the modeling of pancreas, Seg Vol is significantly superior to other baseline methods.
22

---

Ground Truth
3DUX-NET
SwinUNETR
nnU-Net
SegVol
Spleen
Stomach
Figure 13: Visualized spleen and stomach prediction results of 3DUX-NET22 23 SwinUNETR 20) (2 and Vol 4 cases from the split test set. For the consistency and stability of stomach modeling, Seg Vol is significantly better than other methods.
23

---

MedSAM
SAM
SAM-MED20
SAM-MED3D points
SegVol
polints
Aorta
Bladder
Figure 14: Visualized aorta and bladder prediction results MedSAM SAM(bbox) 28), SAM-
MED2D 38), SAM-MED3D 39), SAM(points) and Sol on 4 cases from split test data.
24

---

MedSAM
SAM
SAM-MED2D points/slice
SAM-MED30 points
SAM points
SagVal text bbox
Gall Bladder
Left Kidney
Figure 15: Visualized gall bladder and left kidney prediction results of MedSAM SAM(bbox)[28), SAM-MED2D(39), SAM-MED3D 39], SAM(points) and Seg Vol on 4 cases from split test data.
25

---

MedSAM
SAM-MED20 points/slice
SAM-MED30 points
SegVol bbox
points
Prostato/ Uttrru
Figure 16: Visualized liver and prostate/uterus prediction results of MedSAM SAM(bbox) 28). SAM-MED2D(38) SAM-MED3D [39] SAM(points) (88) and Vol on 4 cases from split test data.
26

---

Table 9: Comparative experiment results of 3DUX-NET, SwinUNETR, nnU-Net, and SegVol on the test set of supervised line-tuning datasets in terms of Dice score. Dice scores are displayed as 'Median values (First quartile, Third quartile)'.
Category
22
SwinUNETR 20
DnU-Net 22
SegVol
Aorta
0,9122 (0.8852, 0.9292) 0.8870 (0.8619, 0.8964) 0.9155 (0.8790. 0.9431) 0.9179 (0.8850, 0.9256)
Colon cancer
0.0773 (0 10,0000, 0.2931) 0.0270 00.0003, 0.2908) 0.3610 (0.0000, 0.6961) 0.7582 (0.6749, 0,7903)
Esophagus
0.7136 (0.6617, 0.7718) 0.6063 (0.5508, 0,6353) 0,7407 (0.6563, 0.8313) 0.7373 (0.7205, 0,8062)
Gallbladder
0.4916 (0.1875, 0.6926) 0.2714 (0. 14.1, 0.567 0.8555 (0.5267. 0.8633) 0,8560 (0.7036, 0.8968)
Inferior vena cava
0.7673 (0.6740, 0.8465)
0.7368 (0.6376, 0,8376) 0.8138 (0.7580, 0.8487) 0.8267 (0.8044, 0.8418)
Left adrenal gland
0.5788 (0.3238, 0.6038)
0,5658 (0.4380, 0.6147) 0.7915 (0.6888, 0.8231) 0.7643 (0.6525, 0,7880)
Left kidney
0,9072 (0.8692, 0,9438) 0:9070 (0. (0.8829, 0,9203) 0.9395 (0.9050, 0:9518) 0.9296 (0.9228, 0,9321)
Liver
0,9316 (0.9074, 0.9462) 0.9374 (0.9110, 0.9531) 0.9276 (0.8614, 0.9597) 0.9560 (0.9437, 0.9685)
Liver tumor
0.7131 (0.5159, 0.8457) 0.6479 (0.2756, 0.7853) 0.7495 (0.6243, 0.8228) 0.7801 (0.7558, 0.8440)
Lung tumor
0,5628 (0.4375, 0,7021) 0.4043 (0.2159, 0,6910) 0.7294 (0.4814, 0.8210) 0.7250 (0.6026, 0,8154)
Pancreas
0,5820 (0.4748, 0.7069) 0.6352 (0.5586, 0.6894) 0.8248 (0.8169. 0.8665) 0.8464 (0. (0.8248, 0.8578)
Portal vein splenic vein
0.7207 (0.6211, 0.7588) 0.6656 (0.5888, 0.6982) 0.7964 (0.7524. 0.8582) 0,7188 (0.7128, 0.7569)
Right adrenal gland
0.5785 (0,5099, 0.6302) 0.5026 (0.2730. 0.5963) 0.7137 (0.7067. 0.7326) 0.6579 (0.6372, 0,7008)
Right kidncy
0.9177 (0.8877, 0,9417) 0.9065 (0.901 0,9289) 0,9432 (0.9207, 0.9504) 0.9227 (0.9157, 0,9295)
Splcen
0.8913 (0.7726, 0.9492)
0.9147 (0.8255, 0,9456) 0.9681 (0.9596, 0.9766) 0,9642 (0.9558, 0.9664)
Stomach
0,7627 (0.6655, 0.8424) 0.7147 (0.6470, 0,8231) 0.8374 (0.6339, 0.9391) 0.9177 (0.9035, 0,9260)
Text+Bbox
Bbox
Text+5 Points
5 Points Text
Dice Score
0.8
DeepLesion3D
Bone Lesion
Pancreas Lesion
Aorta
Bladder
Duodenum
Esophagus
Gall Bladder
Adrenal Gland(L) Left Kidney
Liver
Pancreas
Postcava
Prostate Uterus
Adrenal Gland(R)
Right Kidney
Spleen Stomach
Figure 17: The bar chart illustrates the consistency of Seg Vol's performance across different prompt types on the validation set of AMOS22 44] and (L4
resize and sliding window strategies in 15 organ categories. In the aspect of inference efficiency, the coom-out-zoom-i1 mechanism is also very competitive and quite close to the simple resize method.
Generalization performance of Seg on MRI. We discuss the generalization performance of VegVol on an external MRI dataset. We collect 60 MRI scans annotated with 4 key organ categories dataset and evaluate the generalization ability to unseen modality of Seg Vol. It achieves median Dice scores of 85.70% 80.09%, 80.04%, and 81.46% for liver, spleen, left kidney, and right kidney, respectively. This generalization result demonstrates the robustness of Seg Vol the face of completely unseen modality data. The detailed scores and visualization results presented in Table [2 and Figure 18
from CHAOS 40, 41 42
Few-shot fine-tuning experiment on small datasets. To evaluate the few-shot learning ability of our model, we conduct the few-shot fine-tuning experiment on small datasets, [LARE22[556 (5] (40 training cases) and MSD-spleen(56 (56] (32 training cases). Table demonstrates that 1) fine-tuning Seg Vol on dozens of samples works well on easy datasets such as MSD-spleen, in which the few-shot learning performance close to the joint fine-tuning on all datasets; 2) for challenging datasets such as FLARE22, fine-tuning all datasets can achieve much better performance.
27

---

Comparative experiment results of SAM(Ponnt). SAM(Bbox), SAM-MED2D. SAM-MED3D, MedSAM. and Seg Vol in terms of Dice score. Dice scores are displayed as 'Median values (First quartile, Third quartile)" The three columns the table are, order, the AMOS22 ULS23[74] and SegTHOR (75] datascis.
0.8704 (0.8260, 0.9141) 0.8 (0.6680. 0.8692) 0.3387 (0.2778, 0.4478) 0.9273 (0.9050, 0.9424)
0.9120 (0.8338. 0.9446)
0.5066 (0.4170, 0.5725) 0.3820 (0.2427, 0.4981) 0.3066 (0.2635, 0.3661) 0.7402 (0.6594. 0.7909)
0.5174 (0.3678, 0.6792) 0,3610 (0.2560, 0.5402) 0.7460 (0.6366, 0.8115)
0.7999 (0.7097, 0.8225) 0.5643 (0.3615, 0.7377, 0.6609 (0.5446,, (.7245) 0.8763 (0.8020. 0,9082)
0.5068 (0.3225. 0.6318) 0.4584 0.4544 (0.3104, 0.6267) 0.3766 (0.3321, 0.4541) 0.7295 (0.5119 0.7916)
0.9325 (0.8899, 0.9467) 0.8723 0.8723 (0.7705, 0.9286) 0.7909 (0.7409, (.8199) 0.9489 (0.9389, 0.9585)
0.9641 (0.9547, 0.9701)
0.8295 (0.7734, 0.8711)
0.8384 (0.7909, 0.8684)
0.8557 (0.825 0.8901)
0.6994 (0.6188. 0.7661)
0.9505 (0.9426, 0.9585)
0.9589 (0.9465, 0.9677)
0.4378 (0.3503. 0.5379) 0.9123 (0.8677 0.9369)
0.7473 (0.6817, 11..7.0..000. 0.8063) 0.3258 (0.1325, 0.5707) 0.2386 (0. 1045, 0.4372) 0.7680 (0.7103. 0.8160) 0.7065 (0.6247. 0.7782)
0.6920 (0.6097, 0.7702)
0.7265 (0.6722, 0.7776)
0.3278 (0.2786, 0.3650) 0.8439 (0.8246. 0.8673)
0.7201 (0.6512, 0.7598)
0.8172 (0.7664, 0.8609)
0.1261 (0.1012, 0.1648) 0.8807 (0.8571 0.9044)
Seg Vol
0.8417 (0.7484, 0,9024) 0.4338 (0.2445. 0.7198) 0.6799 (0.4275. 0.7992)
0.6904 (0.5401, 0.8016) 0.8801 (0.8204 0.9321) 0.6137 (0.5783, 0,6479)
0.5391 (0.3 (3.3300, 0,7333) 0.3217 (0.2756. 0.4020)
0.5211 (0.4998. 0.6180)
0.7739 (0.6685. 0.8271)
0.1681 (0.0873, 0.3560) 0.3708 (0.2454, 0.5182) 0.3855 (0.3 0.4710)
0.7851 (0.7506, 0.8227)
0.7038 (0.6609. 0.7766)
0.6896 (0.6128, 0.7530)
0.6561 (0.5857, 0.7144)
0.2196 (0. 1633. 0.2630)
0.8924 (0.8782, 0.9099)
MedSam 29
0.4436 (0.3029, 0.6463) 0.6683 (0.5353, 0.7672)
0.7518 (0.6567. 0.8261) 0.6231 (3.5330. 0.7364)
0.9077 (0.8685, 0.9419) 0.8632 (0.7755, 0.9258)
0.8591 (0.7552. 0.9297)
0.5399 (0.4555. 0.6267) 0.4576 (0.2540, 0.6447)
0. 0.1947 (0.0898, 0.3969) 0.4447 (0. 1481, 0.7026)
0.0675 (0.0471, 0.1237) 0.5579 (0.4911,0.611))) 0.5548 (0.4862, 0.6382) 0.5526 (0.3287, 0.6786)
0.3894 (0. (0.3356. 0.4325) 0,8077 (0.7693, 0.8289) 0,7703 (0.7090. 0.8133)
0.3578 (0. (0.2545, 0.428!) 0.6394 (0.5477, 0.7267)
0,8876 8652, 0.9073) 0.6012 (0.4050, 0.6922) 0.8326 (08059. 0.8536)
0.9147 (0.6688, 0.9545) 0.161 (0. 1161, 0.2218) 0.8306 (0. 7501, 0.8642) 0.8485 (081100 0.891 I)
SAM-MED3D 39
0.5500 (0.4131, 0.6599)
0.5656 (0.5155, 0,6413)
0.9267 (0.8821, 0,9483)
SAM-MED3D 38
0.7267 (0.5213, 0.9350) 0.4362 (0. 1..3991 0.5646)
0.4162 (0. (0.2862. 0. 1.5099) 0.6281 (0. 00.3933. 0.7565)
0.1554 (0. 1039 0.2125) 0.3192 (0. 2552559 0.3886)
0.2917 (0. 10.1019, 0.6169) 0.3541 (0.2167, 0.5540)
0.6161 (0. (0.4809, 0.7200)
0.4222 (0.3417, 0.4995)
0.8274 07.7733. 0.8631)
0.5124 (0.4467, 0.5801)
0.3392 (0.2572, 0.4243)
0.2042 (0.1402, 0.3478) 0.5251 (0.4349, 0.5925)
0.6986 (0. 5430, 0.7522)
0.3642 (7.2766. 0.4491)
0.8215 (0. 7528. 0.8577)
0.6536 (0. 5934, 0.7697)
0.3883 (0. 3051, 00.4713)
0.6671 (0. 5854, 0.7443)
0.2046 (0. 1416, 0.2455)
SAM [28]
0.2831 (0.1756, 0.5198)
0.0555 (0.0276, 0.2347)
0.8405 (0.6844, O. 0.9464)
0.7477 (0.6695, 0.8085)
0.2127 (0.1558, 0.3109)
0.2344 (0.1655, 0.3081)
0.0452 (0.0268, 0.1082)
0.8459 (0. (0.5935, 0.9497)
0.5936 (0 0.4666. 0.7846)
0.4229 (0.34377 0.5479)
0.3686 (0.0855, 0.7680)
0.4461 (0.2349. 0.6676)
0.2744 (0.2289, 0.3488)
0.0348 (0.0250. 0.0459)
0.6695 (0.6180 80, 0.7419)
SAM(Point) 28
Right adrenal gland
Left adrenal gland
Prostate uterus
DeepLesion3l 3D
Pancreas Lesion
Table 10:
Duodenum
Esophagus
Gallbladder
Left kidney
Right kidney
BoneLesion
Esophagus
Category
Bladder
Pancreas
Postcava
Spleen
Stomach
Trachea
Aorta
Liver
Aorta
Heart
28

---

Table 11: Dice score and inference time results of ablation study on zoom-out-zoom-in mechanism.
Category
Mechanism
Dice Score Avg.
Time Per Case Avg.
Resize
0.6375
64 ms
Arota
Zoomn-out-zoomin 0.8853
341
Sliding window
0.8715
3452
Resize
0.3229
65 ms
Bladder
Znon-outtzoommin
0.6530
135 ms
Sliding window
0.5146
3098 ms
Resize
0.2821
64 ms
Duodenum
Zoom-out-zoom-in
0.6414
168
Sliding window
0.5013
3442 ms
Resize
0.1778
62 ms
Esophagus
zooo--nutzzoomin
0.4530
180 ms
Sliding window
0.3372
3186 ms
Resize
0.3558
63 ms
Gall bladder
Zoom-out-zoom-in
0.6830
140 ms
Sliding window
0.5758
3220 ms
Resize
0.0937
64 ms
Left adrenal gland
Zoom-out-zoom-in
0.4542
134 ms
Sliding window
0.2080
2935 ms
Resize
0.6472
64 ms
Left kidney
Zoom-out-7oom-in
0.9362
177 ms
Sliding window
0.9046
3440 ms
Resize
0.8252
62 ms
Liver
zoom-out-zoom-in
0.9593
287 ms
Sliding window
0,9516
3450 ms
Resize
0.3444
65 ms
Pancreas
Zoom-out-zoom-a
0.7292
159 ms
Sliding window
0.6206
3447 ms
Resize
0.5882
63 ms
Postcava
Zoom-oute out-zoom-in
0.7901
296 ms
Sliding window
0.7769
3453
Resize
0.4072
65 ms
Prostate/uterus
Zoom moom--uttzzom-iii
0.6380
136 ms
Sliding window
0.5500
3034 ms
Resize
0.1660
64
Right adrenal gland
7oom-nut-zoom-in
0.5260
141
Sliding window
0.3624
3442
Resize
0.6570
64 ms
Right kidney
Zoom-oun coom-out-zoom-in
0.8909
175 ms
Sliding window
0.9017
3441 ms
Resize
0.6827
89
Spleen
ioom-out-zoom-in
0.8942
199 ms
Sliding window
0.8941
3481 ms
Resize
0.5752
63 ms
Stomach
Zoom-out- coa--ootttzoooooe zoom-in
0,8136
181 ms
Sliding window
0.8238
3446 ms
29

---

Table 12: Generalization experiment results of Seg on the MRI of CHAOS (40
42 dataset in term of Dice score. Dice scores are displayed as 'Median values (First quartile, Third quartile)'.
Method
Liver
Spleen
Left Kidney
Right Kidney
SegVo Vol(5 Points) 0.8091 (0.7376, 0.8554) 0.7496 (0.6990, 0.7872) 0.7216 (0.6125, 0.7869) 0.7174 (0.6052, 0.8090)
Seg Bol 0.8570 (0.8319, 0.8819) 0.8009 (0.7702, 0.8256) 0.8004 (0.7265, 0.8452) 0.8146 (0.7593, 0.8620)
Image
Ground Truth
SegVol
Image
Ground Truth
SegVol
Liver
Spleen
Kidneys
Figure 18: Visualized liver, spleen, and kidney prediction results of Seg cases from MRI SolVol Vol still able to segment these four organs
of CHAOS{0 41 42). For unseen MRI modality, Seg relatively accurately.
D Evaluation Metrics
Each subset of the joint dataset is split into 80% training data and 20% test data. To ensure the absence of any data leaks, the hash value utilized to compare the test set and training set. And in the comparative experiments, the model's parameters are all frozen.
We use the Dice Similarity Coefficient (Dice score) as metric to evaluate the model, which is defined X
as DSC = | [X n Y| is the cardinality of the intersection the predicted segmentation sets X and the ground truth sets Y. [X| and YY| are the cardinalities of sets X and Y respectively. Dice
30

---

score is commonly used metric for evaluating image segmentation tasks. It measures the degree of similarity between predicted segmentation and segmentation, making particularly suitable for evaluating the overlap degree of binary segmentation results.
E
Additional Discussion
We present SegVol, a 3D foundational model for interactive and universal volumetric medical image segmentation. This method has been developed using 90K unlabeled CTs and 25 open-source medical datasets. This results in a universal segmentation tool capable of generating accurate responses for over 200 anatomical targets. Furthermore, Sog demonstrates state-of-the-art volumetric segmentation performance when compared with both traditional task-specific methods [1
[2 (2 23
and the recent SAM-like interactive methods (29] [38 33 (8 in several comparative experiments. Despite its universality and high precision, Seg Vol maintains a simple architecture compared to other
volumetric segmentation methods.
VegVol's capability interactive and precise segmentation makes it a promising clinical aid tool. assist clinicians in identifying and quantifying tumor location, size, and shape changes within patient's body [1] more accurately and rapidly. This precise monitoring aids clinicians in detecting tumor growth trends, assessing treatment effectiveness, and adjusting treatment plans needed. Additionally, clinicians can use SegVol to accurately identify and segment important structures within patient's ss body, such organs, blood vessels, the precise location of tumors and surrounding tissues, using high-resolution 3D images such as CT volumes. These precise segmentation results help clinicians better understand the patient's s anatomical structures, plan surgical pathways, reduce surgical risks, and improve the accuracy and success rate of surgeries 33.
While Seg Vol is capable of understanding momantic-prompe composed of sentences, there remains gap between it and the referring expression segmentation that involves complex semantic information and logical relationships. The establishment of referring expression segmentation model needs more curated data with spatial annotations with text. Our Seg Vol provides foundation for realizing referring segmentation of medical images, and we leave it as future work.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work in Section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
32

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All of the datasets involved in this work are open-source and accessible, which have been summarized Section A The proposed model deseribed in Section (.2 The detailed training algorithm is present in Section The experimental setup is given in Section The trained model and code will be released after the review period.
3.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open Access to Data and Code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All of the datasets involved this work are open-source and accessible, which have been summarized in Section The construction process of data is described in Section The trained model and code will be released after the review period.
2.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps: cc/
public/guides /ooe//odeeeeeeeieeeeee cwr.mmsssooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips http:/cccppbbccccgcccccec. iideeiiiieeeeee des / as onPol PiPoolicyy licyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The detailed experiment setting and information testing data is described in Section 3.1 The trained model and code will be released after the review period.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments conducted for multiple times. The violin plots of 32 segmentation tasks are provided in Figure aanre in Figure Median values, first quartiles, and third quartiles of comparative experiments are present in Table and Table TO
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on computer resources is provided in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttt:::/eeerrissccc//ppppppeeee http:///wwwppp....o//a///cccclca
Question:
Answer: [Yes]
Justification: Research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We contribute a foundation model for universal and interactive volumetric and applications.
medical image segmentation, which can benefit numerous clinical study We do not see any obvious negative societal impact of the proposed method and model. Detailed discussion is provided in the Section and Section
Guidelines:
35

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No model this paper is with high risk for misuse. The collected datasets are all open-source and accessible.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets, used in the paper, are all properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssw woperswithhoos com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All of the datasets involved this work are open-source and accessible, which have been summarized in Section The trained model will be released after reviewing.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "A5JpaXNbqL", "reviewer": "Reviewer_8irr", "summary": "The paper describes the utilization of SegVol, a deep learning model, to segment any organ/tumor/lesion on 3D CT data.\n\n**objectives**\nCreate a universal segmentation model that can segment with :\n-  any type of labeling\n-  good performances on complex tasks\n-  low computational cost (i.e., no sliding window)\n  \n**contributions**\nThe authors contribute to the state of the art in 3 ways :\n- first model for segmenting over 200 anatomical categories\n- support spatial (points/bbox) and semantic prompt (text describing each category)\n- introduction of a zoom-out-zoom-in mechanism to improve the model's efficiency and accuracy\n\n**datasets**\n- 25 open-source segmentation CT datasets + background filtering \n- generation of pseudo-labels with Felzenswalb-Huttenlocher to remove spurious correlation between the datasets\n\n**architecture**\nThe whole architectures is divided into four blocks:\n- image encoder: 3D ViT pretrained using SimMIM on 96K unlabeled CTs + fine-tuning on 6K labeled CTs\n- text encoder: the CLIP model is used to generate a text embedding from the text prompt. The text prompt only consists in the name of the anatomical region to be segmented.\n- prompt encoder: text embedding and spatial prompt (bbox+points) are given to generate the prompt embedding.\n- mask decoder: from the prompt embedding, the segmentation is generated.\n\n**training algorithm**\n- the encoder is pretrained using SimMIM\n- the model is fine-tuned using an algorithm described in appendix B. For each epoch, an iteration of training is made using the ground truth, and a second one using the pseudo masks.\n\n**results**\nExperimental setup is given, as well as a description of the three external datasets used for the evaluation.\n- comparison with SAM-like methods : Table 2 shows the superiority of the method with Dice score > 0.7 on each task. Figure 2 helps to visualize the dice score distribution for each model, on each task. An additional comparison to standard fully supervised segmentation models such as nnUNet, SwinUNetR is provided. Superiority of SegVol is also shown compared to this model on Figure 8 and Table 6.\n- ablation studies\n  - Zoom-out-zoom-in mechanism : Table 3 shows the interest of the method, improving performances and reducing duration compared to sliding window\n  - Scaling up training data : Fig 3.(a) shows that the model performances improve as the number of training data increases.\n  - Prompt impact : Fig 3 (a and b) shows that point prompt is inferior to text prompt, and best prompt combination is bbox+text\n- Case studies\n  - Disambiguation via semantic-prompt: Figure 4 shows that when the spatial prompt is not sufficient to describe the segmenting task, the text prompt adds the complementary information to locate the good anatomical region.\n  - spatial-prompt impact: Figure 5 shows how the points prompt are related to the results. When choosing two points, the category becomes clear, whereas with one point can identify two categories.\n\n**discussion**\n- Scalability : the authors believe that the model have a strong model scalability.\n- Generalizability : Generalization of SegVol to MRI is discussed, with an example given in appendix C showing a strong dice score of 80%.\n- Limitations : only limitation is that text prompt only consider a category, and not a full sentence with logical reasoning.\n- Broader impact : the authors believe that their method is universal and doesn't have any negative impact.", "questions": "**datasets**\nl. 83 : Why is Felzenswalb-Huttenlocher (FH) algorithm to generate pseudo masks ?\nl. 81 : How is it supposed to remove spurious correlation between datasets ?\n\n**architecture**\nl. 96 : The architecture principle is described, but some details are lacking such as the number of layers of each network, the number of heads in the ViT, etc.\nl. 133 : why is the computational cost reduced using the zoom-in-zoom-out principle compared to sliding window? clarify this point.\nl. 140 : how are prompts generated from the coarse segmentation masks? clarify if the prompts are generated thanks to the prompt encoder as a bbox, or points.\n\n**results**\nl. 169 : Could you indicate the training time (SimMIM and finetuning) of the model?\nl. 169 : Why did you choose these 3 specific datasets as external test sets?\n\n**discussion**\nl. 257 : Why is data salability a good thing ? Why is this part in the discussion? The data scalability of the model is not discussed or compared with other models. If not discussed, move to the conclusion.", "limitations": "Only one limitation is addressed, considering the text prompt which cannot be a logical sentence at the moment.\nHowever, the amount of data required to train the model is not discussed. Would a few-shot finetuning method work on small datasets (10 samples)?", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- **originality** : the concept of SegVol is not original itself. However, the addition of the zoom-in-zoom-out principle and the combination of bbox + text + points prompts is new in this kind of models.\n- **quality** : the work is done rigorously, respecting a scientific reasoning all along the article. \n- **clarity** : the paper lacks clarity as there are two parts difficult to understand.\n  - 1) The description of the training algorithm is not made in the text, only the steps are written in appendix B.\n  - 2) The hyperparameters regarding the model's dimension (number of layer, ViT dimension) is not explicited.\n- **significance** : The results are impactful for the community of medical image segmentation, and could have plenty of applications. The property of segmenting multiple organs, from multiple entirely different datasets is remarkable.", "weaknesses": "The only weakness regards the clarity of the training algorithm, which is hard to understand even with appendix B and training steps.\nThis point could be improved by describing the steps in the appendix with a paragraph, or maybe with another diagram that follows the exact same steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your high recognition of our paper and the detailed feedback. We answer your questions as follows.\n\n> **Q1: Clarity of the training algorithm. Describing the steps with a paragraph or diagram.**\n\nThank you for the suggestion. We will provide the detailed text description of the Training Algorithm in the final version. Here, we give a simple diagram, Figure 1 in the attached PDF, and a brief description for your reference. Specifically, each case (training sample) consists of an Image *x*, a Ground Truth(GT) Mask Set *Y*, and a Pseudo Mask Set *Z*. The training loss of each sample consists of the ground-truth loss and the pseudo loss. The ground-truth loss is computed by inputting the image, the ground-truth mask (label) and the sampled prompt into the model, while the pseudo loss is computed by inputting the image, the pseudo label and the fixed prompt into the model. Finally, the model is updated by minimizing the weighted sum of the two losses.\n\n\n> **Q2: Why is FH algorithm to generate pseudo masks? How is it supposed to remove spurious correlation between datasets?**\n\nThe FH algorithm is an unsupervised segmentation method that can generate segmentation masks purely based on the voxel similarity of regions in CT scans. In the beginning, we also tried several candidates, including *morphological_chan_vese*, *morphological_geodesic_active_contour*, *quickshift*, *slic*, etc. Finally, we found the FH algorithm performs best, and thus chose it to generate pseudo masks.\n\nSince in most datasets, the 3D CTs are annotated with only a few segmentation categories, which causes a spurious correlation between the labeled categories and the specific dataset. Using pseudo masks can supplement unlabeled categories in a dataset, therefore relieving the spurious correlation.\n\n> **Q3: Some details are lacking such as the number of layers of each network, the number of heads in the ViT, etc.**\n\nThanks for reminding, we will add the network details in the paper. Here are main hyper-parameters of the ViT in SegVol: \n\npatch_size=(4, 16, 16), num_layers=12, num_heads=12, hidden_size=768.\n\nThe code and model weights of SegVol will be released after the review period.\n\n> **Q4: Why is the computational cost reduced using the zoom-out-zoom-in principle compared to the sliding window?**\n\nThe traditional sliding window method requires scanning the entire 3D-CT, processing *hundreds* of windows. In contrast, the proposed zoom-out-zoom-in mechanism only requires one global inference of 3D-CT and scanning the ROI with *dozens* of windows. \n\n> **Q5: How are prompts generated from the coarse segmentation masks?**\n\nThe process of prompts generated from the coarse segmentation masks is the same as that of prompts generated from ground truth masks, where point prompts are created in or beside the targets, bbox prompts are formed near the target boundaries, and text prompts are derived from category names. The detailed strategies are presented in Page 4 Sec 2.3.\n\n> **Q6: Could you indicate the training time (SimMIM and finetuning)?**\n\nSimMIM pre-training time: ~20$\\times$8 GPU hours on NVIDIA A100-SXM4-40GB.\n\nFinetuning time: ~300$\\times$8 GPU hours on NVIDIA A100-SXM4-40GB.\n\n> **Q7:** **Why did you choose these 3 specific datasets as external test sets?**\n\nAMOS22 is a popular dataset for medical image segmentation. Many works are trained and evaluated on it. The validation set of AMOS22, as a representative dataset of abdominal organs, has a large number of samples and includes 15 major abdominal organs.\n\nThe ULS23 dataset is a novel large-scale lesion segmentation dataset with thousands of cases covering various lesions, which is a challenging benchmark of lesion segmentation.\n\nSegTHOR dataset focuses on the thorax, which can be a supplement of AMOS22 (abdomen) and ULS23 (lesions). On the whole, the three datasets cover most of the segmentation tasks of organs and lesions in the thorax and abdomen.\n\n> **Q8: Why is data salability a good thing? Why is this part in the discussion?** \n\nThe scaling law of foundation models has been verified in multiple CV and NLP tasks. We achieved the success of scaling law in 3D medical segmentation by the design of universal prompts and pseudo masks for joint learning on datasets with inconsistent annotations. We provided a preliminary experiment of the data scaling law in Figure 3 (a) Page 7, which shows that 1) the performance improves significantly with more training data, 2) and SegVol has not yet reached its ceiling if more training data is provided. We will provide more detailed discussion and comparison to other methods in the final version.\n\n> **Q9: The amount of data required to train the model (SegVol) is not discussed. Would a few-shot finetuning method work on small datasets (10 samples)?**\n\nFigure 3 (a) in Page 7 demonstrates the relationship between the model performance and the amount of finetuning training data. Generally speaking, more training data leads to better performance. \n\nWe conducted the few-shot finetuning experiment on small datasets, FLARE22 (40 training cases) and MSD-spleen (32 training cases), to study the few-shot learning ability of SegVol. The table below demonstrates that 1) finetuning SegVol on dozens of samples works well on easy datasets such as MSD-spleen, in which the few-shot performance is close to the joint finetuning on all datasets; 2) for challenging datasets such as FLARE22, finetuning on all datasets can achieves much better performance than few-shot finetuning.\n\n| Avg. Dice Score | 100  epochs | 200 epochs | 300 epochs | 400 epochs | 500 epochs | SegVol* |\n| :-------------- | :---------: | :--------: | :--------: | :--------: | :--------: | :-----: |\n| FLARE22         |   0.0463    |   0.4028   |   0.4926   |   0.5617   |   0.5567   | 0.8822  |\n| MSD-spleen      |   0.7566    |   0.7866   |   0.9433   |   0.9454   |   0.9471   | 0.9597  |\n\n*Note: **SegVol\\*** represents the model finetuned on all datasets.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9iFJZH3lo", "reviewer": "Reviewer_rczM", "summary": "This paper proposes a 3D foundation segmentation model, named SegVol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes,  this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. Besides, a zoom-out-zoom-in mechanism is designed to facilitate efficient and precise inference on volumetric images.", "questions": "Answer the questions in Weakness Section", "limitations": "Not Applicable", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Collect and process 25 public volumetric medical segmentation datasets, encompassing over 200 anatomical categories. The pseudo label is introduced to relieve the spurious correlation in the training data.\n(2) Implement massive 3D pre-training on 96K CT volumes and supervised fine-tuning on the  6k labeled datasets.\n(3) Support spatial-prompt, semantic-prompt, and combined-prompt segmentation, achieving  high-precision segmentation and semantic disambiguation.\n(4) Design a zoom-out-zoom-in mechanism that significantly reduces the computational cost, meanwhile preserving precise segmentation.", "weaknesses": "(1) As shown in Appendix Table 5, the numbers of trainset volumes is very dunbalanced， which will affect the performance of the proposed method. How the author address the above issue? They should give further analysis and validation.\n(2) For unseen anatomical categories, what is the segmentation effect of SegVol? Provide relevant experiments for further explanation.\n(3) What are the complexity、number of parameters and running speed of the SegVol, and provide a comparison with existing SOTA methods.\n(4) Minor: The reference format should be unified，for example,[8]、[43]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Below we answer the specific questions.\n\n> **Q1: As shown in Appendix Table 5, the number of train-set volumes is very unbalanced, which will affect the performance of the proposed method.** \n\nIt is supposed that the unbalanced training set may cause poor performance on the minor categories/domains in some tasks. However, this problem does not matter in our work. We studied this problem by training SegVol on those small sub-datasets and comparing its performance to that trained on the joint of all datasets. Specifically, BTCV, MSD-spleen, and FLARE22 are used as the minor sub-datasets (domains), which have 24, 32, and 40 training samples respectively, and account for only 0.5%, 0.7%, and 0.9% of the whole training set. The following table shows that SegVol trained on the whole dataset, which consists of 25 sub-datasets, achieves much better performances than those trained on the small sub-datasets respectively. The results indicate that SegVol, as a foundation model, performs well on minor sub-datasets (domains) and does not suffer from the problem of unbalanced training data. We will provide more analysis in the final version.\n\n| **Avg. Dice Score**                 |    BTCV    | MSD-spleen |  FLARE22   |\n| :---------------------------------- | :--------: | :--------: | :--------: |\n| SegVol trained on small sub-dataset |   0.5195   |   0.9471   |   0.5567   |\n| SegVol trained on the whole dataset | **0.8058** | **0.9597** | **0.8822** |\n\n> **Q2: For unseen anatomical categories, what is the segmentation effect of SegVol?**\n\nIn Page 6 Table 2 and Page 7 Line 202, we presented the experiments on ULS23, where most lesion categories are unseen for SegVol. The unseen categories include most abdominal lesions, all bone lesions, some lung lesions, and all mediastinal lesions. For unseen categories, we use spatial prompts and general semantic prompts, e.g., ‘lesion’ or ‘tumor’, to drive the model. The results show that SegVol can achieve good segmentation performance (Dice: 0.7046) on these unseen categories. \n\nAfter the review phase, we will host an online running model for users to test various cases from unseen categories.\n\n> **Q3: What are the complexity, number of parameters and running speed of the SegVol?**\n\nThe following table shows the Total Parameters, the average Multiply-Accumulates(MACs), and the average Time required to process a case of the different methods. The comparison indicates that our method takes less computational cost while achieving much better performance.\n\nNote that when calculating MACs Per Case and Time Per Case, the slice-by-slice calculation of the 2D method and the scanning process of the 3D method are accumulated respectively. SAM-MED3D only processes volume with a size of $128\\times128\\times128$. The experiments are implemented on the validation set of AMOS22, and the setting is the same as that in Sec. 3.2.\n\n| **Method** | **Total Parameters** | **Avg. MACs Per Case↓** | **Avg. Time Per Case (s)↓** | **Avg. Dice Score↑** |\n| :--------- | :------------------: | :---------------------: | :-------------------------: | :------------------: |\n| SAM        |         94M          |         1.3e+13         |           2.1764            |        0.5271        |\n| MedSAM     |         94M          |         1.3e+13         |           2.1886            |        0.5371        |\n| SAM-MED2D  |         271M         |         2.3e+12         |           3.5547            |    *0.6668*     |\n| SAM-MED3D  |         101M         |       **1.0e+11**       |         **0.1768**          |        0.6200        |\n| SegVol     |         181M         |     *6.7e+11*      |        *0.3283*        |      **0.8593**      |\n\n\n> **Q4: Minor: The reference format should be unified.**\n\nThanks for your reminder. We will unify the reference format in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYm2yp9dth", "reviewer": "Reviewer_Ykum", "summary": "In this paper, the authors proposed SegVol, a 3D foundation segmentation model for medical images. This model supports universal and interactive volumetric medical image segmentation of more than 200 anatomical categories. This model is also well-designed with spatial and semantic prompts.\n\nIn the inference stage, the authors designed a zoom-out-zoom-in mechanism to enable efficient and precise segmentation results\n\nTheir extensive experiments show their method surpasses other SAM-like interactive segmentation methods.", "questions": "I would appreciate it if the authors could answer my questions.\n\n1. Figure 2, the Violin plots seem confused and it is not cited in the paper. Can the authors explain more about this figure?\n\n2. Will some of the 25 public volumetric medical segmentation datasets share the same data? Can the authors provide more detailed information about the datasets?\n\n3. How do the authors process the collected data with inconsistent annotations? For example, different hospitals will have different annotations of the Aorta. Will this influence the results?", "limitations": "The authors adequately addressed their limitations in the manuscript as well as the appendix.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Interesting concepts. The idea of integrating spatial and semantic prompts is interesting and achieves prospective results.\n\nPaper clarity. The paper is overall well-written and structured. I also enjoyed the quality of the figures which help understanding the method.\n\nGood results. The method achieves SoTA results compared with the SAM-like interactive segmentation methods.", "weaknesses": "I had a hard time finding weaknesses in the paper. Those I find are either nitpick or more directions for future work. \n\nI put the weaknesses in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and kind words. We answer your questions as follows.\n\n> **Q1: Figure 2, the violin plots seem confused and it is not cited in the paper.**\n\nFigure 2 is cited on Page 7 Line 205: *‘We visualize the Dice score distributions of all methods in all the tasks as violin plots, depicted in Figure 2.’* \n\nFigure 2 on Page 5 and Table 2 on Page 6 describe the results of the same experiments from two different perspectives. Table 2 presents the precise quantitative results (the median value), while the violin plots in Figure 2 illustrate the distributions of the results for intuitive comparison. \n\n> **Q2: Will some of the 25 public volumetric medical segmentation datasets share the same data? Can the authors provide more detailed information about the datasets?**\n\nThere are no duplicate cases in our processed 25 public datasets. We used the hash code to validate all cases (image-label pair) during the data processing phase. It was found that the LiTS dataset contains many duplicate cases which also exist in other datasets, thus the whole LiTS dataset was removed. Besides, one duplicate case was found and removed in the remaining 25 public datasets. \n\nFor more detailed information about these open-source datasets, please refer to their official websites which are provided in Page 17 Table 5.\n\n> **Q3:** **How do the authors process the collected data with inconsistent annotations?**\n\nIt is an important advantage of SegVol that the model is compatible with inconsistent but reasonable annotations. There is no need for additional data processing. For example, the whole ‘kidney’ is annotated in one hospital, while more specifically ‘left kidney’ and ‘right kidney’ are annotated in other hospitals. SegVol can learn to comprehend such inconsistent annotations and generalize well to users’ rough or precise semantic+spatial prompts in the testing phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers’ constructive comments and valuable suggestions.

We are glad to see that our paper received high praise from all reviewers. Especially, **Reviewer 8irr** finds that ‘*the results are impactful for the community of medical image segmentation, and could have plenty of applications.*’ **Reviewer Ykum** thinks that ‘*I had a hard time finding weaknesses in the paper. Those I find are either nitpick or more directions for future work.*’

We address the specific questions from reviewers as follows.

Meta-review: The paper introduces SegVol, a universal deep learning model designed for segmenting any organ, tumor, or lesion in 3D CT data. SegVol leverages a substantial amount of training data and incorporates both spatial and semantic prompts to enhance segmentation accuracy. The model is noted for its innovative zoom-out-zoom-in mechanism, which improves efficiency and performance without the need for a sliding window approach.

Strengths :
While SegVol itself is not entirely novel, its combination of zoom-in-zoom-out principles and prompt types is innovative.
The work is rigorous and scientifically sound.
The paper is generally clear, though it lacks detailed descriptions of the training algorithm and model hyperparameters.
The model's ability to segment multiple organs from diverse datasets is highly impactful.

Weaknesses :
The training steps are hard to understand and could benefit from additional explanation and diagrams.
The unbalanced volume of training data could affect performance.
The segmentation performance on unseen categories needs further validation.
The complexity, parameter count, and running speed of SegVol need to be compared with existing SOTA methods.

Recommendations :
The training steps could be better presented even with Appendix B
Only one limitation is mentioned regarding the text prompt's current capacity.
The discussion of the required amount of training data required could be addressed, and the potential for few-shot fine-tuning could be explored as well.
Decision: Accept (spotlight)