SegVol: Universal and Interactive Volumetric Medical Image Segmentation

Authors: yuxin  du
fan  bai
tiejun  huang
bo  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:07:23

Abstract: Precise image segmentation provides clinical study with instructive information. Despite the remarkable progress achieved in medical image segmentation, there is still an absence of a 3D foundation segmentation model that can segment a wide range of anatomical categories with easy user interaction. In this paper, we propose a 3D foundation segmentation model, named SegVol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes, this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. To facilitate efficient and precise inference on volumetric images, we design a zoom-out-zoom-in mechanism. Extensive experiments on 22 anatomical segmentation tasks verify that SegVol outperforms the competitors in 19 tasks, with improvements up to 37.24\% compared to the runner-up methods. We demonstrate the effectiveness and importance of specific designs by ablation study. We expect this foundation model can promote the development of volumetric medical image analysis. The model and code are publicly available at https://github.com/BAAI-DCAI/SegVol.

Main paper: 
 introduction: 
 Volumetric medical segmentation, involving extracting 3D regions of interest, such as organs, lesions, and tissues, plays a pivotal role in medical image analysis by accurately modeling the 3D structural information of the human body from volumetric medical images such as CT or MRI. The accurate segmentation can benefit numerous clinical applications including tumors monitoring [1, 2] , surgical planning [3, 4] , disease diagnosis [5] , therapy optimization [6, 7] , etc.Compared to 2D medical image segmentation [8, 9, 10, 11, 12, 13, 14, 15, 16, 17] , volumetric image segmentation is notably more challenging due to the labor-intensive annotation and resourceconsuming computation. Recently, the research of volumetric medical image segmentation has garnered substantial attention, leading to a series of advancements [18, 19, 20, 21, 22, 23] . However, existing volumetric medical segmentation methods have several key limitations which prevent their application in challenging tasks, e.g., liver tumor or colon cancer segmentation [24, 25, 26, 27] , and real-world tasks, e.g., human-interactive segmentation [28, 29, 30, 31, 32] .Firstly, the publicly available volumetric medical image datasets usually consist of a small number of mask annotations from a few varying categories. Due to the different label spaces, the traditional task-specific segmentation models trained on one dataset have difficulty in generalizing to others. For example, the CT-ORG dataset [33, 34, 24, 35] contains the 'lungs' category, while this category is split into two sub-classes and named 'left lung' and 'right lung' in the LUNA16 dataset [36] . Hence, a universal segmentation model has to understand the semantics of anatomical categories. Secondly, traditional segmentation models have inferior performance when segmenting complex structures, such as tumors and cysts [37] . This is because these models are trained on insufficient data and are also not able to leverage the spatial information through user interaction. Thirdly, previous solutions are computationally expensive in the inference process. They typically employ a sliding window to infer the whole volumetric input. This strategy is not only time-consuming but also short-sighted, as the sliding window contains only local information. Recently, there have been some works [29, 38, 39] that introduce spatial-prompt into medical image segmentation, shown in Table 1 . However, most of them lack the ability to process the 3D input directly and naturally, and none of them is able to understand the semantics of anatomical categories.In this paper, we propose the first foundation model for volumetric medical image segmentation -SegVol. The proposed model enables universal and interactive 3D segmentation of more than 200 anatomical categories, supporting both spatial and semantic prompts. SegVol can also be driven by the combination of multi-prompt, like 'bounding box+text' or 'point+text' prompts, achieving highprecision segmentation and semantic disambiguation. To enable efficient and precise segmentation of volumetric images, we develop a zoom-out-zoom-in mechanism that enables the model to be efficient and precise. We evaluate the proposed SegVol on 22 volumetric medical image segmentation tasks and the results demonstrate our method surpasses other SAM-like interactive segmentation methods [28, 38, 39, 29] by a large margin. Extensive case studies and ablation experiments are also carried out to prove the advantages of SegVol and the effectiveness of the zoom-out-zoom-in mechanism and multi-prompt combination.We summarize our key contributions as follows:1. Collect and process 25 public volumetric medical segmentation datasets, encompassing over 200 anatomical categories. The pseudo label is introduced to relieve the spurious correlation in the training data. 2. Implement massive 3D pre-training on 96K CT volumes and supervised fine-tuning on the 6k labeled datasets.3. Support spatial-prompt, semantic-prompt, and combined-prompt segmentation, achieving high-precision segmentation and semantic disambiguation.4. Design a zoom-out-zoom-in mechanism that significantly reduces the computational cost, meanwhile preserving precise segmentation. 
dataset construction: 
 One of the main challenges of training a universal volumetric medical segmentation model is the absence of large-scale publicly available volumetric medical data, especially CTs with segmentation annotations. Doing our utmost, we collected 25 open-source segmentation CT datasets, including CHAOS [40, 41, 42] , HaN-Seg [43] , AMOS22 [44] , AbdomenCT-1k [45] , KiTS23 [46] , KiPA22 [47, 48, 49, 50] , KiTS19 [51] , BTCV [52] , Pancreas-CT [53, 54, 35] , 3D-IRCADB [55] , FLARE22 [56, 57] , TotalSegmentator [58] , CT-ORG [33, 34, 24, 35] , VerSe19, VerSe20 [59, 60, 61] , SLIVER07 [62] , QUBIQ [63] , six MSD datasets [56] , LUNA16 [36] , and WORD [64] . Their detailed information and availability are shown in the Section A. These CTs originate from various medical institutions, captured by different machines with varying parameter settings and scanning regions. To standardize these datasets, we use the mean voxel value of each volume to filter the background and then perform normalization on the foreground voxels.Volumetric segmentation datasets suffer from the notorious problem of partial labels. Most of these datasets have annotations of only a few segmentation targets, e.g., several organs. Therefore, the deep models may learn the spurious correlation between datasets and segmentation targets, and thus produce inferior results during the inference phase. To relieve this problem, we introduce the pseudo labels by utilizing the Felzenswalb-Huttenlocher (FH) [65] algorithm to generate pseudo masks for each CT scan. Pseudo masks can supplement unlabeled categories in a dataset, therefore relieving the spurious correlation problem. To restrain the noise and numerous tiny masks in pseudo labels, we employ the following strategies: 1) The pseudo masks are replaced with ground truth masks when applicable. 2) We filter out tiny structures smaller than 1‰ of the whole volume size. 3) Each mask is refined by dilation and erosion operations.Motivated by the recent advance in 2D nature image segmentation, Segment Anything (SAM) [28] , we design a novel model for interactive and universal volumetric medical image segmentation, named, SegVol. The model is illustrated in Figure 1 . SegVol supports three types of prompts for interactive segmentation: 'bounding box(bbox)' prompt, including the coordinates of two diagonal vertices; 'point' prompt, composed of a set of positive and negative points; and 'text' prompt, such as 'liver' or 'cervical spine C2'. The model consists of four modules: image encoder, text encoder, prompt encoder, and mask decoder.We employ 3D ViT (Vision Transformer) [66, 67] as the image encoder, which exhibits remarkable advantages over convolutional models [68] when pre-trained on large-scale datasets. The 3D ViT structure is designed as follows: patch size= (4, 16, 16) , layers number=12, heads number=12, hidden size=768. We first pre-train 3D ViT using SimMIM algorithm [69] on the collected 96K CTs, and then conduct further supervised fine-tuning on the 6K CTs with 150K labeled segmentation masks.One of the main limitations of traditional segmentation models is that the models learn dataset-specific labels encoded as integers which cannot generalized to unseen datasets or tasks, preventing their real-world applications. We enable universal segmentation across datasets by leveraging the text encoder from CLIP model [70] to encode the input text prompt, as CLIP [70] has been trained to align image and text embeddings on web-scale image-text pairs. Given a word or phrase as the text prompt, we complete it using the template 'A computerized tomography of a [text prompt]' [71] and then encode it into text embedding. The off-the-shelf text encoder is frozen during training due to the limited text data in CT datasets. Following SAM [28] , we obtain the spatial-prompt embedding using positional encoding [72] on point and bbox prompt.After obtaining the image embedding and prompt embedding, we input them into the mask decoder and predict the mask. We use self-attention and cross-attention in two directions to fuse the image embedding and prompt embedding, and then employ the transposed convolutions and interpolation operations to generate masks. Since text embedding is the key to universal segmentation and it is also challenging to learn the correlation between text and volumetric regions, we enhance the text information by introducing a parallel text input branch beside the joint prompt embedding.SegVol accepts multiple types of prompts, including individual point, bbox, and text prompts, and also their combinations. To make full use of the segmentation training data, we generate kinds of prompts for each datum and construct kinds of prompt-mask data pairs for training.The point prompt is built from ground truth or pseudo masks, consisting of three kinds of points, namely, positive point, negative point, and ignored point. Positive point means that it is within the target mask region, while negative points are those outside. Ignored points are utilized to ensure a uniform length of the point prompts for input completion. Notably, these ignored points are not considered by the model.The bbox prompt is generated based on the ground truth or pseudo masks, integrated with random jitter to enhance the model's robustness. When generating the bbox prompt for some pseudo mask, the bbox may also cover other masks due to the irregular 3D shapes. To address this problem, we compute the Intersection over Union (IoU) between the generated bbox and the included pseudo masks. Any mask with an IoU greater than 0.9 will also be integrated and considered as part of the target mask corresponding to this bbox prompt.The text prompts are constructed based on their category names. As pseudo masks produced by the unsupervised FH algorithm [65] do not have the semantic information, we only use point and bbox prompts for training on masks of pseudo labels.SAM-like interaction with large-volume images is laborious for users, especially in the scene where the sliding window has to be used due to the limited view of those 3D models. To provide users with an easy SAM-like interface, we design a zoom-out-zoom-in mechanism, which is efficient and precise, consisting of zoom-out-zoom-in inference and multi-size training. As demonstrated in Figure 1 , the zoom-out process involves resizing a volumetric image, which is input into the model with user prompts to generate a coarse segmentation mask. Then, the Region of Interest (ROI) from the original image is cropped for zoom-in analysis. In the zoom-in process, a sliding window is used to perform precise inference driven by prompts generated from the coarse segmentation mask. After that, the ROI prediction mask will be back-filled to the coarse segmentation mask to finish the final prediction. Besides, multi-size training involves augmenting the input data by resizing CTs for the zoom-out view and cropping them into cubes for the zoom-in view. The zoom-out-zoom-in mechanism realizes the computational cost reduction meanwhile producing precise segmentation of the ROI.We apply SimMIM algorithm [69] to pre-train the image encoder of SegVol with the masked image modeling loss L pre-training (θ IE ; D 1 ). The loss function is as follows:EQUATIONwhere θ IE is the parameter set of SegVol's image encoder. a, b ∈ R D×H×W are the input voxel values and predicted values, respectively. M denotes the set of masked voxels, Ω(•) is the number of elements, and D 1 is the pre-training dataset.We combine the Binary Cross-Entropy (BCE) loss and Dice loss as the supervised fine-tuning loss function L fine-tuning (θ; D 2 ) to train the model with trainable parameters θ (text encoder frozen). D 2 is the supervised fine-tuning dataset and x, y ∈ R D×H×W are the predicted mask and ground-truth mask, respectively. F(•, θ) is the forward function of SegVol. The loss function is as follows:EQUATIONEQUATIONEQUATIONThe detailed fine-tuning algorithm of SegVol is presented in Section B. 
experiments: 
 In this section, we conduct extensive experiments on 22 volumetric medical image segmentation tasks to compare SegVol with other SAM-like medical image segmentation methods [28, 38, 39, 29] . Ablation studies are also carried out to prove the effectiveness of the zoom-out-zoom-in mechanism and provide more insights about dataset scale and multi-prompt combination. Detailed case studies are conducted to discuss the disambiguation ability of semantic-prompt and the capability of identifying the segmentation results with spatial-prompt.During the pre-training, we follow SimMIM algorithm [69] to train the 3D ViT encoder of SegVol on the collected 96K CTs for 2000 epochs. In the supervised fine-tuning stage, we train SegVol (with the text encoder frozen) on the labeled 25 volumetric medical image segmentation datasets for 270 epochs with batch size 32 and input size (32, 256, 256) , using AdamW optimizer [73] . SimMIM pre-training takes about 20 × 8 GPU hours, while fine-tuning takes about 300 × 8 GPU hours. All the above training process is implemented on 8 NVIDIA A100-SXM4-40GB. Three external datasets [44, 74, 75] and 20% testing data preserved from 25 collected datasets are used in the following experiments.Several efforts have been made to construct a SAM-like interactive medical image segmentation model. However, some of these works, such as MedSAM [29] and SAM-MED2D [38] , focus on 2D tasks and cannot process 3D input directly. The other 3D-based methods, such as SAM-MED3D [39] , only support small cropped input and do not support semantic-prompt segmentation, which are still far from building a comprehensive foundation model for volumetric medical image analysis.Competitors and configures. In this experiment, MedSAM [29] and SAM(bounding box) [28] use bounding box prompts. SAM(5 clicks) [28] , SAM-MED2D [38] and SAM-MED3D [39] use point prompts and a five-step correction procedure, which means that the point prompt in each step will be given according to the previous-step output and ground truth, rather than giving all at once. In In addition, we compare SegVol with traditional task-specific segmentation models, e.g., 3DUX-NET [23] , SwinUNETR [20] , and nnU-Net [22] , in Section C, though the direct comparison is unsuitable due to the different settings and objectives.Testing data. To compare with these SAM-like interactive segmentation models, we evaluate the models on 1,778 cases from the validation set of AMOS22 [44] , the whole novel annotated set of Uni- versal Lesion Segmentation Challenge 23(ULS23) [74] , and the released labeled set of SegTHOR [75] .The validation set of AMOS22 contains 120 cases annotated with 15 major organs. The novel annotated ULS23 dataset is composed of three subsets, namely, DeepLesion3D, Radboudumc Bone, and Radboudumc Pancreas. The DeepLesion3D subset contains 200 abdominal lesions, 100 bone lesions, 50 kidney lesions, 50 liver lesions, 100 lung lesions, 100 mediastinal lesions, and 150 assorted lesions cases. There are 744 bone lesion cases in the Radboudumc Bone subset and 124 pancreas lesion cases in the Radboudumc Pancreas subset. The 40 cases from SegTHOR, which are contoured manually by an experienced radiotherapist, focus on the heart, trachea, aorta, and esophagus that surround the tumor and must be preserved from irradiations during radiotherapy.Quantitative results. The quantitative results of comparative experiments are shown in Table 2 , which verify our method is the best in most of the tasks including both lesions and organs, compared to other SAM-like interactive models [28, 38, 39, 29] . Specifically, our method outperforms the second-ranked SAM-MED2D on the AMOS22 dataset by a significant improvement of 19.25% (average Dice score). On the SegTHOR dataset, our method surpasses the runner-up -SAM-MED3D by an average Dice score improvement of 4.28%. The ULS23 dataset, characterized by small patchlike masks, presents a unique challenge. In this scenario, SegVol still exhibits good performance, comparable to MedSAM, which excels in using bbox prompts for segmenting small objects. We visualize the Dice score distributions of all methods in all the tasks as violin plots, depicted in Figure 2 . More detailed results and visualization are present in Section C.Zoom-out-zoom-in mechanism. One of the key designs of SegVol is the zoom-out-zoom-in mechanism. We compare it with the intuitive resize strategy and the popular sliding window algorithm on the split 20% test data, 48 cases covering 15 major organs with a variety of sizes, belonging to the AMOS22 [44] dataset. Two evaluation dimensions, i.e., performance (Dice score) and inference time cost (per case), are compared, as shown in Table 3 . The zoom-out-zoom-in mechanism achieves the best average Dice score and a very competitive inference speed compared to the simple resize strategy. The reason for computational cost reduction is that the traditional sliding window method requires scanning the entire 3D CT and processing thousands of windows. In contrast, the proposed zoom-out-zoom-in mechanism only requires one global inference of 3D CT and then scanning the ROI with dozens of windows. Detailed experiment results are shown in Section C.Scaling up training data. The success of scaling up training data has been witnessed in multiple computer vision tasks [28, 70] . We conduct an ablation study to investigate the importance of scaling up training images and masks. The split 20% test data of BTCV dataset [52] , which includes 13 main organs, is set as an anchor to evaluate the model trained separately on 1, 2, and 8 datasets for 500 epochs, as well as the final model trained on 25 datasets. The detailed results are shown in Figure 3 (a). As a lightweight model, the performance of SegVol is weak when only one dataset is used. However, with the increase of training data, the Dice score increases rapidly, especially in the text prompt setting. The results indicate that our method is scalable and better performance can be achieved if more training data is available.Multi-prompt combination. As a universal model, our approach achieves precise segmentation for over 200 organs, tissues, and lesions using both spatial and semantic prompts. In Figure 3 (b), we quantitatively analyze the mutually supportive relationship between semantic-prompt and spatial-prompt in 19 segmentation tasks of the 20% split test data. On the one hand, spatial-prompt allows the model to locate the specific part in the 3D space. According to Figure 3 (b), the average Dice score of the 'bbox+text' prompt is boosted by 5.85% compared to the 'text' prompt on average. On the other hand, semantic-prompt clarifies the reference to the anatomical structure, eliminating the ambiguity of spatial-prompt and the plausible masks of multiple categories. This is reflected in Figure 3 (b) as the average Dice score of 'point+text' prompts is 4.62% higher than using 'point' prompts alone. Spatial and semantic prompts mutually support each other, ultimately endowing the model with powerful segmentation capabilities.Disambiguation via semantic-prompt. It is a notorious problem in interactive segmentation that one spatial-prompt may correspond to multiple plausible outputs [28] . As illustrated in the images on the top left in Figure 4 , three of them correspond to three anatomical concepts, namely, kidney tumor, left kidney, and the whole kidneys, while they are all plausible to the same point prompt. Similarly, in the bottom left three images, the bounding box selects the region of the liver. However, liver tumors, hepatic vessels, and the liver itself are also plausible target structures. In these cases, SAM chooses to return multiple masks to match different levels of plausible results. Unlike SAM's solution, we use semantic-prompt to clarify the targets. As shown in Figure 4 , the captions below the images are the text prompts, and the masks in the images are the predictions of SegVol, which show that semantic-prompt can effectively disambiguate the spatial-prompt.Identifying the spatial-prompt segmentation. Furthermore, we study the capability of SegVol to identify the semantic category of the spatial-prompt results. Figure 5 reveals that SegVol can give accurate semantic categories based on the spatial-prompt results. In the top left image in Figure 5 , the spatial-prompt on the liver results in a 0.997 prediction score for the liver. The top right image in the sub-figure shows if the spatial-prompt is the point on the liver tumor, SegVol will output a 0.619 prediction score for the tumor category and a 0.339 prediction score for the liver based on the spatial relationship of liver tumor and liver. We implement this identification experiment by decoding the semantic prompts from a category set. The softmax function is applied to the decoding results to get the prediction probabilities of different categories. The probabilities on the initial predicted mask, driven by the spatial-prompt, are used to calculate the final classification result. 
discussion: 
 Scalability. The scaling law of foundation models has been verified in multiple CV and NLP tasks. Since SegVol uses a transformer-based architecture and self-supervised pre-training algorithm, it has strong data and architecture scalability. In this work, we achieve the success of scaling law in 3D medical segmentation by the design of universal prompts and pseudo masks for joint learning on datasets with inconsistent annotations. The ablation study of scaling up training data shows that 1) the performance improves significantly with more training data in the 3D segmentation task, 2) SegVol has not yet reached its ceiling if more training data is provided. We believe the performance of SegVol can be continuously improved when more data and computational resources are used. Limitations. Although SegVol shows remarkable semantic-prompt segmentation performance, there still remains gap between it and the referring volumetric segmentation. A promising solution is to construct the referring segmentation data with diverse semantic and spatial prompts, and then train SegVol on it. We leave it as the future work. More discussions can be found in Section E.Broader impact. We contribute a foundation model for universal and interactive volumetric medical image segmentation, which can benefit numerous clinical study and applications. As a foundational research work, we do not see any obvious negative societal impact of the proposed method and model. 
conclusion: 
 In this paper, we propose SegVol, a universal and interactive volumetric medical image segmentation model, supporting both spatial-prompt and semantic-prompt segmentation of more than 200 anatomical categories. We construct a large-scale dataset, which consists of 90K unlabeled CTs and 25 open-source medical datasets, to train the foundation model. We design the zoom-out-zoom-in mechanism to facilitate efficient and precise inference in the region of interest. Extensive experiments on 22 segmentation tasks demonstrate the outstanding performance of our method. Detailed ablation studies are also carried out to prove the effectiveness of the zoom-out-zoom-in mechanism, dataset scale, and multi-prompt combination strategy. As a foundation model, we believe that SegVol will advance the volumetric medical segmentation and benefit numerous downstream tasks. Besides, we add a reinforcement branch for semantic-prompt in the mask decoder. We further compute a similarity matrix between the up-scaled embedding from the transposed convolution output and the text embedding. The element-wise multiplication of the similarity matrix with the mask prediction is applied before interpolation, after which the model outputs the masks. pt spatial , pt semantic , ⇐ prompt_generate(y i )7:l gt ⇐ 0 8:# Loop for possible prompt combination types of ground truth mask. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We contribute a foundation model for universal and interactive volumetric medical image segmentation, which can benefit numerous clinical study and applications. We do not see any obvious negative societal impact of the proposed method and model. Detailed discussion is provided in the Section 4 and Section E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No model in this paper is with a high risk for misuse. The collected datasets are all open-source and accessible.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets, used in the paper, are all properly credited. The license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: All of the datasets involved in this work are open-source and accessible, which have been summarized in Section A. The trained model will be released after reviewing.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: for p ⇐ 1 to n do 10:# Choose prompt combination type. l pseudo ⇐ 0 18:# Loop for several pseudo masks.for p ⇐ 1 to n do 20:# Random select a pseudo mask of this case for training.21:z p ⇐ RandomSelect(Z x , [1, m]) 22:pt spatial ⇐ prompt_generate(z p ) Comparative experiments to compare with task-specific segmentation models. Task-specific segmentation models mainly fall into two architectures, CNN-based models and Transformer-based models. We conduct comparative experiments with representative CNN-based models i.e. 3DUX-Net [23] and nnU-Net [22] , and representative Transformer-based models i.e. SwinUNETR [20] . We conduct additional comparative experiments on the split 20% test set of the datasets. 10 segmentation tasks are selected from BTCV [52] and MSD-spleen [56] datasets, which focus on organ segmentation, and from MSD-lung, MSD-colon, and MSD-liver datasets, which focus on lesion segmentation. We train task-specific segmentation models on each dataset individually for each method.The quantitative experimental results are summarized in Figure 9 . Generally speaking, SegVol, jointly trained on 25 datasets, outperforms traditional task-specific segmentation models trained on a single dataset. Compared to these strong baselines, SegVol exhibits a narrower distribution of Dice scores across the eight tasks, indicating its robustness and good generalization ability. This mainly owes to the massive knowledge learned from diverse samples of the same categories but different datasets. SegVol depicts excellent performance on lesion tasks which are more challenging in semantic understanding and spatial locating. We present a detailed comparison to nnU-Net [22] on lesion tasks. As shown in Table 7 , the average Dice score of SegVol is 14.76% higher than that of nnU-Net for lesion tasks. We visualize the prediction results of the two methods in Figure 10 , which intuitively show that SegVol performs more precise segmentation of the tumors than nnU-Net. The detailed scores and visualization results are presented in Table 9 and Figure 11 We analyze that there are mainly three factors that make SegVol more powerful than traditional taskspecific models: 1) Massive generative pre-training on unlabeled data endows SegVol with a complete understanding of the volumetric structures and the discriminative feature representations, which is much superior to learning from a small number of samples. 2) Learning from joint datasets with semantic-prompt makes SegVol generalize better to unseen data and categories. For instance, SegVol can learn from both the 'left kidney' and 'kidney' categories based on their semantic correlation, while traditional task-specific models treat the two categories independently. 3) SegVol can be prompted with (spatial) points/bboxes, which provide a precise spatial reference, and (semantic) texts, which disambiguate the overlap of multiple categories in the same space. In contrast, traditional methods are not able to understand semantics. This ability enables SegVol to perform better than traditional methods in challenging tasks, e.g., segmenting lesions. Figure 11 : Visualized aorta and left kidney prediction results of 3DUX-NET [23] , SwinUNETR [20] , nnU-Net [22] and SegVol on 4 cases from the split test set. For the integrality of aorta and left kidney structure modeling, SegVol significantly outperforms 3DUX-NET and SwinUNETR and is comparable to nnU-Net.Liver Pancreas Figure 12: Visualized liver and pancreas prediction results of 3DUX-NET [23] , SwinUNETR [20] , nnU-Net [22] and SegVol on 4 cases from the split test set. For the modeling of pancreas, SegVol is significantly superior to other baseline methods.Ground TruthSpleen Stomach Figure 13: Visualized spleen and stomach prediction results of 3DUX-NET [23] , SwinUNETR [20] , nnU-Net [22] and SegVol on 4 cases from the split test set. For the consistency and stability of stomach modeling, SegVol is significantly better than other methods. [29] , SAM(bbox) [28] , SAM-MED2D [38] , SAM-MED3D [39] , SAM(points) [28] and SegVol on 4 cases from split test data. [29] , SAM(bbox) [28] , SAM-MED2D [38] , SAM-MED3D [39] , SAM(points) [28] and SegVol on 4 cases from split test data. [29] , SAM(bbox) [28] , SAM-MED2D [38] , SAM-MED3D [39] , SAM(points) [28] and SegVol on 4 cases from split test data. Figure 17 : The bar chart illustrates the consistency of SegVol's performance across different prompt types on the validation set of AMOS22 [44] and ULS23 [74] .resize and sliding window strategies in 15 organ categories. In the aspect of inference efficiency, the zoom-out-zoom-in mechanism is also very competitive and quite close to the simple resize method.Generalization performance of SegVol on MRI. We discuss the generalization performance of SegVol on an external MRI dataset. We collect 60 MRI scans annotated with 4 key organ categories from CHAOS [40, 41, 42] dataset and evaluate the generalization ability to unseen modality of SegVol. It achieves median Dice scores of 85.70%, 80.09%, 80.04%, and 81.46% for liver, spleen, left kidney, and right kidney, respectively. This generalization result demonstrates the robustness of SegVol in the face of completely unseen modality data. The detailed scores and visualization results are presented in Table 12 and Figure 18 .Few-shot fine-tuning experiment on small datasets. To evaluate the few-shot learning ability of our model, we conduct the few-shot fine-tuning experiment on small datasets, FLARE22 [56, 57] (40 training cases) and MSD-spleen [56] (32 training cases). Table 8 demonstrates that 1) fine-tuning SegVol on dozens of samples works well on easy datasets such as MSD-spleen, in which the few-shot learning performance is close to the joint fine-tuning on all datasets; 2) for challenging datasets such as FLARE22, fine-tuning on all datasets can achieve much better performance. score is a commonly used metric for evaluating image segmentation tasks. It measures the degree of similarity between predicted segmentation and true segmentation, making it particularly suitable for evaluating the overlap degree of binary segmentation results.We present SegVol, a 3D foundational model for interactive and universal volumetric medical image segmentation. This method has been developed using 90K unlabeled CTs and 25 open-source medical datasets. This results in a universal segmentation tool capable of generating accurate responses for over 200 anatomical targets. Furthermore, SegVol demonstrates state-of-the-art volumetric segmentation performance when compared with both traditional task-specific methods [20, 21, 22, 23] and the recent SAM-like interactive methods [29, 38, 39, 28] in several comparative experiments. Despite its universality and high precision, SegVol maintains a simple architecture compared to other volumetric segmentation methods.SegVol's capability of interactive and precise segmentation makes it a promising clinical aid tool. It can assist clinicians in identifying and quantifying tumor location, size, and shape changes within a patient's body [1] more accurately and rapidly. This precise monitoring aids clinicians in detecting tumor growth trends, assessing treatment effectiveness, and adjusting treatment plans as needed. Additionally, clinicians can use SegVol to accurately identify and segment important structures within a patient's body, such as organs, blood vessels, or the precise location of tumors and surrounding tissues, using high-resolution 3D images such as CT volumes. These precise segmentation results help clinicians better understand the patient's anatomical structures, plan surgical pathways, reduce surgical risks, and improve the accuracy and success rate of surgeries [3] .While SegVol is capable of understanding semantic-prompt composed of sentences, there remains a gap between it and the referring expression segmentation that involves complex semantic information and logical relationships. The establishment of a referring expression segmentation model needs more curated data with spatial annotations with text. Our SegVol provides a foundation for realizing referring segmentation of medical images, and we leave it as future work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All of the datasets involved in this work are open-source and accessible, which have been summarized in Section A. The construction process of data is described in Section 2.1. The trained model and code will be released after the review period. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The information on computer resources is provided in Section 3.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: Research conducted in the paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces SegVol, a universal deep learning model designed for segmenting any organ, tumor, or lesion in 3D CT data. SegVol leverages a substantial amount of training data and incorporates both spatial and semantic prompts to enhance segmentation accuracy. The model is noted for its innovative zoom-out-zoom-in mechanism, which improves efficiency and performance without the need for a sliding window approach.

Strengths :
While SegVol itself is not entirely novel, its combination of zoom-in-zoom-out principles and prompt types is innovative.
The work is rigorous and scientifically sound.
The paper is generally clear, though it lacks detailed descriptions of the training algorithm and model hyperparameters.
The model's ability to segment multiple organs from diverse datasets is highly impactful.

Weaknesses :
The training steps are hard to understand and could benefit from additional explanation and diagrams.
The unbalanced volume of training data could affect performance.
The segmentation performance on unseen categories needs further validation.
The complexity, parameter count, and running speed of SegVol need to be compared with existing SOTA methods.

Recommendations :
The training steps could be better presented even with Appendix B
Only one limitation is mentioned regarding the text prompt's current capacity.
The discussion of the required amount of training data required could be addressed, and the potential for few-shot fine-tuning could be explored as well.

Reviews: review: paper_summary: paper_summary: The paper describes the utilization of SegVol, a deep learning model, to segment any organ/tumor/lesion on 3D CT data.

**objectives**
Create a universal segmentation model that can segment with :
-  any type of labeling
-  good performances on complex tasks
-  low computational cost (i.e., no sliding window)
  
**contributions**
The authors contribute to the state of the art in 3 ways :
- first model for segmenting over 200 anatomical categories
- support spatial (points/bbox) and semantic prompt (text describing each category)
- introduction of a zoom-out-zoom-in mechanism to improve the model's efficiency and accuracy

**datasets**
- 25 open-source segmentation CT datasets + background filtering 
- generation of pseudo-labels with Felzenswalb-Huttenlocher to remove spurious correlation between the datasets

**architecture**
The whole architectures is divided into four blocks:
- image encoder: 3D ViT pretrained using SimMIM on 96K unlabeled CTs + fine-tuning on 6K labeled CTs
- text encoder: the CLIP model is used to generate a text embedding from the text prompt. The text prompt only consists in the name of the anatomical region to be segmented.
- prompt encoder: text embedding and spatial prompt (bbox+points) are given to generate the prompt embedding.
- mask decoder: from the prompt embedding, the segmentation is generated.

**training algorithm**
- the encoder is pretrained using SimMIM
- the model is fine-tuned using an algorithm described in appendix B. For each epoch, an iteration of training is made using the ground truth, and a second one using the pseudo masks.

**results**
Experimental setup is given, as well as a description of the three external datasets used for the evaluation.
- comparison with SAM-like methods : Table 2 shows the superiority of the method with Dice score > 0.7 on each task. Figure 2 helps to visualize the dice score distribution for each model, on each task. An additional comparison to standard fully supervised segmentation models such as nnUNet, SwinUNetR is provided. Superiority of SegVol is also shown compared to this model on Figure 8 and Table 6.
- ablation studies
  - Zoom-out-zoom-in mechanism : Table 3 shows the interest of the method, improving performances and reducing duration compared to sliding window
  - Scaling up training data : Fig 3.(a) shows that the model performances improve as the number of training data increases.
  - Prompt impact : Fig 3 (a and b) shows that point prompt is inferior to text prompt, and best prompt combination is bbox+text
- Case studies
  - Disambiguation via semantic-prompt: Figure 4 shows that when the spatial prompt is not sufficient to describe the segmenting task, the text prompt adds the complementary information to locate the good anatomical region.
  - spatial-prompt impact: Figure 5 shows how the points prompt are related to the results. When choosing two points, the category becomes clear, whereas with one point can identify two categories.

**discussion**
- Scalability : the authors believe that the model have a strong model scalability.
- Generalizability : Generalization of SegVol to MRI is discussed, with an example given in appendix C showing a strong dice score of 80%.
- Limitations : only limitation is that text prompt only consider a category, and not a full sentence with logical reasoning.
- Broader impact : the authors believe that their method is universal and doesn't have any negative impact.
strength_weakness: strength_weakness: The only weakness regards the clarity of the training algorithm, which is hard to understand even with appendix B and training steps.
This point could be improved by describing the steps in the appendix with a paragraph, or maybe with another diagram that follows the exact same steps.
questions: questions: **datasets**
l. 83 : Why is Felzenswalb-Huttenlocher (FH) algorithm to generate pseudo masks ?
l. 81 : How is it supposed to remove spurious correlation between datasets ?

**architecture**
l. 96 : The architecture principle is described, but some details are lacking such as the number of layers of each network, the number of heads in the ViT, etc.
l. 133 : why is the computational cost reduced using the zoom-in-zoom-out principle compared to sliding window? clarify this point.
l. 140 : how are prompts generated from the coarse segmentation masks? clarify if the prompts are generated thanks to the prompt encoder as a bbox, or points.

**results**
l. 169 : Could you indicate the training time (SimMIM and finetuning) of the model?
l. 169 : Why did you choose these 3 specific datasets as external test sets?

**discussion**
l. 257 : Why is data salability a good thing ? Why is this part in the discussion? The data scalability of the model is not discussed or compared with other models. If not discussed, move to the conclusion.
limitations: limitations: Only one limitation is addressed, considering the text prompt which cannot be a logical sentence at the moment.
However, the amount of data required to train the model is not discussed. Would a few-shot finetuning method work on small datasets (10 samples)?
paper_summary: paper_summary: This paper proposes a 3D foundation segmentation model, named SegVol, supporting universal and interactive volumetric medical image segmentation. By scaling up training data to 90K unlabeled Computed Tomography (CT) volumes and 6K labeled CT volumes,  this foundation model supports the segmentation of over 200 anatomical categories using semantic and spatial prompts. Besides, a zoom-out-zoom-in mechanism is designed to facilitate efficient and precise inference on volumetric images.
strength_weakness: strength_weakness: (1) As shown in Appendix Table 5, the numbers of trainset volumes is very dunbalanced， which will affect the performance of the proposed method. How the author address the above issue? They should give further analysis and validation.
(2) For unseen anatomical categories, what is the segmentation effect of SegVol? Provide relevant experiments for further explanation.
(3) What are the complexity、number of parameters and running speed of the SegVol, and provide a comparison with existing SOTA methods.
(4) Minor: The reference format should be unified，for example,[8]、[43]
questions: questions: Answer the questions in Weakness Section
limitations: limitations: Not Applicable
paper_summary: paper_summary: In this paper, the authors proposed SegVol, a 3D foundation segmentation model for medical images. This model supports universal and interactive volumetric medical image segmentation of more than 200 anatomical categories. This model is also well-designed with spatial and semantic prompts.

In the inference stage, the authors designed a zoom-out-zoom-in mechanism to enable efficient and precise segmentation results

Their extensive experiments show their method surpasses other SAM-like interactive segmentation methods.
strength_weakness: strength_weakness: I had a hard time finding weaknesses in the paper. Those I find are either nitpick or more directions for future work. 

I put the weaknesses in the questions section.
questions: questions: I would appreciate it if the authors could answer my questions.

1. Figure 2, the Violin plots seem confused and it is not cited in the paper. Can the authors explain more about this figure?

2. Will some of the 25 public volumetric medical segmentation datasets share the same data? Can the authors provide more detailed information about the datasets?

3. How do the authors process the collected data with inconsistent annotations? For example, different hospitals will have different annotations of the Aorta. Will this influence the results?
limitations: limitations: The authors adequately addressed their limitations in the manuscript as well as the appendix.

Comments: comment: Thank you for your feedback. We will polish the final version based on your suggestions.
Thank you for the further feedback. We will polish the final version based on your suggestions.
Thank you for your detailed answers to each of my interrogations.
All the questions have been answered by the authors.

Q1 : the algorithm is clearer with this small text, it should be included in the article.
Q2 : the choice of FH is then arbitrary, it should be specified in the article.
Q3 : For the network details, maybe a results of a torchinfo summary, or something in this idea could be added in appendix so that every hyperparameters could be seen visually (it would be be a bonus, but optional).
Q9 : The results of joint finetuning compared to few-shot finetuning is relevent, and could be at least mentionned.

I will maintain my grading, considering the clarity (presentation) will be improved, but is not yet done.
I appreciate the author's detailed response. Most of my concerns have been addressed. I will maintain my original score.
Dear Reviewer rczM,

We hope our response addresses your questions. Please feel free to reach out if you have any further inquiries. We would greatly appreciate it if you could consider raising your rating.

Thank you very much!