AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks
Authors: Chongyu Qu
Tiezheng Zhang
Hualin Qiao
Jie Liu
Yucheng Tang
Alan Yuille
Zongwei Zhou
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Organ Segmentation
Active Learning
Interactive Segmentation
Medical Image Analysis
TL;DR: We introduce AbdomenAtlas-8K, a substantial multi-organ dataset with the spleen, liver, kidneys, stomach, gallbladder, pancreas, aorta, and IVC annotated in 8,448 CT volumes, totaling 3.2 million CT slices.

Abstract: Annotating medical images, particularly for organ segmentation, is laborious and time-consuming. For example, annotating an abdominal organ requires an estimated rate of 30-60 minutes per CT volume based on the expertise of an annotator and the size, visibility, and complexity of the organ. Therefore, publicly available datasets for multi-organ segmentation are often limited in data size and organ diversity. This paper proposes an active learning procedure to expedite the annotation process for organ segmentation and creates the largest multi-organ dataset (by far) with the spleen, liver, kidneys, stomach, gallbladder, pancreas, aorta, and IVC annotated in 8,448 CT volumes, equating to 3.2 million slices. The conventional annotation methods would take an experienced annotator up to 1,600 weeks (or roughly 30.8 years) to complete this task. In contrast, our annotation procedure has accomplished this task in three weeks (based on an 8-hour workday, five days a week) while maintaining a similar or even better annotation quality. This achievement is attributed to three unique properties of our method: (1) label bias reduction using multiple pre-trained segmentation models, (2) effective error detection in the model predictions, and (3) attention guidance for annotators to make corrections on the most salient errors. Furthermore, we summarize the taxonomy of common errors made by AI algorithms and annotators. This allows for continuous improvement of AI and annotations, significantly reducing the annotation costs required to create large-scale datasets for a wider variety of medical imaging tasks. Code and dataset are available at https://github.com/MrGiovanni/AbdomenAtlas

AbdomenAtlas-8K: Annottttn 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks
Chongyu Qu
Tiezheng Zhang' Alan L. Yuille'
Hualin Qiao²
Jie Liu
Yucheng Tang*
Zongwei Zhou
Johns Hopkins University Rutgers University
City University of Hong Kong "NVIDIA
Code Data: htttp:://githii
Abstract
Annotating medical images, particularly for organ segmentation, is laborious and time-consuming. For example, annotating an abdominal organ requires an estimated rate of 30-60 minutes per CT volume based on the expertise of an annotator and the size, visibility, and complexity of the organ. Therefore, publicly available datasets for multi-organ segmentation are often limited in data size and organ diversity. This paper proposes an active learning procedure to expedite the annotation process for organ segmentation and creates the largest multi-organ dataset (by far) with the spleen, liver, kidneys, stomach, gallbladder, pancreas, aorta, and IVC annotated in 8,448 CT volumes, equating to 3.2 million slices. The conventional annotation methods would take an experienced annotator up to 1,600 weeks (or roughly 30.8 years) to complete this task. In contrast, our annotation procedure has accomplished this task in three weeks (based on an 8-hour workday, five days week) while maintaining a similar or even better annotation quality, This achievement is attributed to three unique properties of our method: (1) label bias reduction using multiple pre-trained segmentation models, (2) effective error detection in the model predictions, and (3) attention guidance for annotators to make corrections on the most salient errors. Furthermore, we summarize the taxonomy common emros made by Al algorithms and annotators. This allows for continuous improvement of AI and annotations, significantly reducing the annotation costs required create large-scale datasets for wider variety of medical imaging tasks.
Introduction
Medical segmentation is rapidly advancing task that plays a vital role in diagnosing, treating, and radiotherapy planning [37, 10, 32, 103, 40]. Building datasets of substantial number of annotated medical images is critical for training and testing artificial intelligence (AI) models! [104, 52, 70]. However, medical datasets carefully annotated by annotator are infeasible create at huge scale using conventional annotation methods [65] because performing per- voxel annotations is expensive and time-consumin [89, 55, 53]. As result, publicly available datasets for multi-orgat segmentation are often limited in data size (a few hundred) and organ diversity? as reviewed in Figure ((a).
Corresponding author: Zongwei Zhou (ZZHOU82 @ JILEDU)
'While some of the abdominal organs can be automatically segmented by state-of-the-arr AI models with fairly high accuracy, many other organs and anatomical structures remain suboptimal unknown [55]. 'ToaalSegmentator [89] annotated a diverse set of classes for 1,024 CT volumes, but majority of volumes were largely annotated only by pre- pre-trained nnU-Nets [37]. As result, their annotations are highly biased to the specific architecture (see $5); also, their procedure does not consider inconsistency across different Al models.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

AbdomanAtics-9K (JHU)
lotalSegmentaor 4.0)
AbdomenCT-11 BY-NC-SA)
AMDS 4Y00A440)
WORD (GNU GPL 9.0)
CT-ORG (CC 8.0)
KIS (CC BY-NC-SA 4.0)
UTS (CC BY-SA 4.0)
CHAOS (CC BY-SA
Pancreas CT (CC aY 3.0)
BTCV (CC BY 4.0)
3D-IRCADo (CC BY-NC-ND 4.0)
(a)
2000
4000
Number of CT scans
5000
8000
10000
10a
102
10a
10
spleen
liver
kidney
stomach
gall bladder
pancreas
aorta
postcava
(b)
Figure I: (a) An overview of public datasets. AbdomenAtlas stands out from other datasets
due to its large number of annotated CT volumes. We have reviewed dataset names (and licenses). (b) Volume distribution of eight organs. The significant variations within and across organs presented in Abdomen MadomenAAlas-S5 Atlas-8K present challenges for the multi-organ segmentation and the generalizability of models to different domains. More comparisons can be found in Appendix Table and Figure 9.
There is a pressing need to expedite the annotating procedure. The latest endeavors to construct fully annotated dataset remained ask radiologists manually annotate each and every missing label [58, 36, 41, 78, 72, 9, 85, 21, 55, -9e same annotating procedure used a decade ago [20] or even earlier [64, 17]. Such a procedure extremely costly, particularly for medical images, and will be time-consuming to create large-scale dataset for every medical specialty [69, 90, 13, 83, 65]. To improve the annotation efficiency, active learning has been widely explored by combining the radiologist's 's competence and the computer's capability [106, 107, 71, 8, 11j. However, most studies the active learning literature have been retrospective in nature, in which the annotating procedure was simulated by simply retrieving labels for the data without physically involving the radiologists in the loop to create/revise the labels [105, 66, 87, 94]. In contrast, our study is proactive, not only proposing novel active learning procedure but also implementing it to actually construct large dataset of 8,000 fully annotated CT volumes within a very short span time, leveraging the synergy between medical professionals and Al algorithms in practice.
To overcome the deficiency of the conventional annotation method which involves manually annotating each volume. slice, and voxel we propose an efficient method to enable rapid organ annotation across enormous CT volumes. The efficiency has been demonstrated on 15 datasets and 8,448 abdominal CT volumes (1.2 TB in total). Our method enables high-quality, per-voxel annotations for eight organs and anatomical structures in all the CT volumes in three weeks (5 days per week; 8 hours per day), justified in $3.3. The constructed dataset is named AedomenAtlas-Be and its detailed statistics can be found in Figure 1. As a comparison, the conventional annotation methods [45, 26, 29, 51, 6] would take up to 1,600 weeks (30.8 years) to complete such task..
Our key innovation that rather than time-consuming annotating organs voxel by voxel, we leverage existing data and incomplete labels from an assembly of 16 public datasets. Al models are trained on the labeled part of data and generate predictions for large number of unlabeled parts of data. We then actively revise the model predictions by only selecting the most salient part of the regions for annotators to correct. Moreover, our study provides taxonomy of common errors made by AI algorithms and annotators. This taxonomy minimizes error duplication and augments data diversity, ensuring the sustainability of continuous revision of AI algorithms and organ annotations. Our novel active learning procedure involves only one trained annotator, three Al models, and commercial software (Pair"). ). After the procedure, annotators confirm the annotation of all the CT volumes by visual inspection. A large-scale, but private, dataset [92, 93] is used for external validation.
Trained annotators annotate abdominal organs at rate of 30 60-60 minutes per organ per three- ditee-dimessional CT volume reported by Park et al. [65]. We have also conducted reader study to estimate the annotation time for each CT volume our dataset, detailed Appendix Tahle
We purchased license from Pair. Alternatively, we also used open-source MONAI Label for annotation.

---

In summary, we made two major contributions, Firstly, AbdomenAtlas-SK was composite dataset that unified datasets from 26 different hospitals worldwide. In total, over 60.6 x 10 voxels were annotated in comparison with 4.3 x 10 voxels annotated in the public datasets. We scaled up the organ annotation by a factor of 15 and released the masks for 5,195 of the 8,448 CT volumes. This large-scale, multi-center dataset can also impact downstream clinical tasks such as surgery, treatment, abdomen atlas, and anomaly detection. Secondly, the proposed active learning procedure can generate an attention map to highlight the regions to be revised by radiologists, reducing the annotation time from 30.8 years to three weeks and accelerating the annotation process by an impressive factor of 533, This strategy can quickly scale up annotations for creating many other medical imaging datasets.
2 Related Work
Large dataset construction. Kirillov et al. [43] created a huge natural image dataset of 1B masks and 11M images, but it lacks semantic information, and its efficacy limited when applied to 3D volumetric medical images [52, 34]. Our AbdomenAtlas-8Kt containing 8,448 CT volumes with per-voxel annotated eight abdominal organs, the largest annotated CT dataset at the time this paper is written. We hereby review the existing public datasets that contained over 500 CT volumes with per-voxel annotated organs [23]. For example, AMOS [40], TotalSegmentator [89], and AbdomenCT-
IK [55] provided 500, 1,024, and 1,1 12 annotated CT volumes, respectively. Both TotalSegmentator and AMOS derived their data from a single country, with the former reflecting the Central European population from Switzerland and the latter representing the East Asian population from China. In comparison, AbdomenAtlas collected and assembled from least 26 different hospitals worldwide. While AbdomenCT-1KK sourced data from 12 hospitals, AbdomenAtlas volumes (8,448 vs. 1,000) and twice the variety of annotated organs (8 vs. 4). Concurrently, actively expanding the classes covered by Abdomen AttoomnnAAlllaee Atlas-8K. Starting with the set of 104 classes found in TotalSegmentator, we aim to significantly diversify the range of classes covered.
presented a greater data diversity because the CT volumes were
contained approximately eight times the CT
Active learning for segmentation. Uncertainty and diversity are key criteria in active learning. Uncertainty-base criteria assess the value of annotating data point based on the uncertainty (e.g.. entropy) of AI predictions [18, 19, 56, 75, 5, 63, 11]. On the other hand, diversity- criteria aim to select unannotated samples that differ from each other and from those already annotated [48, 25, 44, 59, 81, 82, 76]. For additional active learning methods, we refer the reader to comprehensive literature reviews [83, 62, 31, 71]; but these methods face computational complexity challenges with segmentation tasks and large unannotated data pools. To overcome this, we summarized typical errors made by humans and computers. Our active learning procedure considered the anatomical priors, uncertainty in Al prediction, and data diversity, and, importantly, pivoted a prospective application of active learning rather than retrospective studies. Moreover, the derived criteria can generate an attention map, pinpointing areas necessitating revision, thereby enabling precise detection of high-risk prediction errors (evidenced in Table 1). Consequently, our strategy markedly diminished the workload and annotation time for annotators by a factor of 533.
AbdomenAtlas-8K
Overview. We propose an active learning procedure comprising two components: error detection from AI predictions ($3.1) and manual revision performed by radiologists review and edit the most significant errors detected ($3.2). By repeatedly implementing these two components, it is possible to expedite the creation of fully annotated datasets for multi-organ semantic segmentation. Finally, describe the data construction strategies ($3.3). In this work, we have applied our approach to 8,448 CT volumes in portal (44%), arterial (37%), pre- (16%), and post-contrast (3%) phases.
3.1 Label Error Detection Revealed by Attention Maps
Figure 2 shows the process to generate attention maps for eight target organs. The attention maps can localize the potential error regions for human annotators to review and edit Al predictions.
(1) Inconsistency. To quantify inconsistency, we calculate the standard deviation of the soft predic- tions produced by multiple AI architectures, including Swin UNETR, nnU-Net, and U-Net. Regions
3

---

un
mancicas
Scan
inconsistency
Uncertainty
Overlap
Attention Map
Figure 2: Attention map generation. The criteria inconsistency, uncertainty, and overlap refer regions where model predictions diverge, exhibit high entropy values, and where multiple organ predictions overlap, respectively. The attention map visualizes a combination of these regions, drawing radiologists' attention to where Al predictions might falter. standard color scheme helps in highlighting regions that merit closer review and revision. More examples are in Appendix Figure 11.
with high standard deviation indicate higher inconsistency and may require further revision.
(P pi.e)
Inconsistency 1,c
(I)
N
where the subseript represents class of our eight target organs. For each voxel i, Ph, represents the soft prediction value obtained from the n-th Al architecture of class at that voxel's 's index i, ranging from 0 to fic represents the average prediction value obtained by combining the results of three AI architectures at the same voxel index. In this study, there are three Al architectures, so N is equal three. Then, the Inconsistency; value is determined by the standard deviation of the soft prediction values from the three AI architectures.
(2) Uncertainty. To estimate the degree of certainty linked with the Al prediction of eight target organs, we determine the entropy of the soft predictions for each organ. Regions of higher entropy values suggest diminished confidence and increased ambiguity, potentially escalating the chances of encountering prediction errors within that specific area [98], which may necessitate further revision.
2 Pl. log(p?.c)
Uncertainty.co
N
The Uncertainty,cc is averaged over different Al architectures (N 3).
(3) Overlap. The overlap organ prediction can indicate potential errors. If voxel is predicted to be part of both the liver and kidneys, even without ground truth, we can reasonably forecast prediction mistake. We use the following measure to detect organ overlap in predictions.
[1
if p.. > 0.5 and = pi,eg 0.5
Overlap,
otherwise
We generate pseudo labels by applying threshold of 0.5 to the probability values. Pseudo labels refer to organ labels predicted by Al models without any additional revision or validation by human annotators. The overlap value, denoted as Overlap, is determined based on the following criteria: the prediction value for class e exceeds the threshold of 0.5 for at least one AI architecture and there exists prediction value not belonging class that exceeds 0.5 for the same voxel index i, then the overlap value is set to 1; otherwise, is set to 0.
As result, an attention map is generated to help annotators quickly locate regions that require revision or confirmation. We combine the inconsistency, uncertainty, and overlapping regions to produce the attention map. Consequently, a higher Attention,, value in the 3D attention map indicates a greater risk prediction error for that voxel.
Attention Inconsistency,.. Uncertainty,.. + Overlap,.
(4)
To assess the attention map, we identified error regions which is the summation of all false positive (FP) and false negative (FN) areas between the ground truth annotations (available in JHH [92]) and the pseudo labels predicted by AI. We then compare the attention maps with the error regions and calculate the sensitivity and precision, reported in 1.

---

3.2 Active Learning Procedure
Algorithm. Our active learning procedure has eight steps. ① Train an AI model from scratch, denoted as Mo. using 2, 100 CT volumes from 16 partially labeled public datasets. This takes approximately 40 hours. Direct test the current model (c.g.. My) on all 8,448 CT volumes segment eight organs. This takes around 12 hours. @ Compute organ-wiss attention maps for each CT volume using the criteria of inconsistency, uncertainty, and overlap ($3.1), 1), which highlight the regions that potentially have prediction errors and require human revision. @ Compile priority list sorted by the sum of attention maps. The larger the attention map is, the more urgent the CT volume requires revision. @ Ask the annotators review the top 5% (analyzed from Figure 4) CT volumes from the list and revise the pseudo labels guided by the attention maps. @ Reassemble the annotation of each revised CT volume based on the label priority (detailed in the next paragraph). @ Fine-tune the current model (M,) using the reassembled annotation to obtain MM+1. annotators confirm that the CT volume the top of the prioritized list in step 4 does not need further revision, suggesting that AI predictions of the most important CT volumes have minimal errors.
Repeat steps 2 until the
Technical details in @: label priority. In the assembly process, the uimost priority is given to the original annotations supplied by each public dataset. Subsequently, we assign secondary priority to the revised labels from our annotators. The pseudo labels, generated by Al models, are accorded the lowest priority. Based on this label priority, we reassemble the labels for each CT volume. This involves three different scenarios: inherited from the original labels of the public dataset if available, revised (if our annotator confirms that the pseudo labels have errors), and unchanged (if our annotator confirms that the pseudo labels predicted by the AI are correct). Figure displays examples of the reassembled annotations from AedoomnnAtlas-888
Figure 3: AblomenAtlas-8K annotations. The annotations for our eight target organs (opaque) have been revised by our annotator or inherited from the original annotations of the partially labeled public datasets. The remaining organs and tumors (transparent) can used for future research.
Pancroas-CC CT
CHAD6
LITS
KITS
WORD
AbdomonCT-1K
Attonion Sac
2030
CT-GRG
MSD Task07 Pancress
MSD Taek00 Vessel
MSD Task09 Spleen
MSD Task06 Lung
MSD Task10 Colon
200
190
200
CT Volume Index
Figure 4: Attention size distribution. The y-axis denotes the attention size, the sum of Equation over eight classes; each point corresponds distinct CT volume. larger attention size implies greater need for revision in various regions. While most CT volumes exhibit small attention size, few notable outliers (marked in red) stand out. These outliers are of high priority for revision by human experts. According to the figure, the ratio of outliers is about 5% (highlighted in red). The 5% estimated by the plot and also related to the budget of human revision for each Step in the active learning procedure. It is essential to emphasize that roughly 5% of CT volumes within each dataset are highly likely to contain predicted errors, requiring further revision by our annotator.
Figure illustrates the attention size of CT volumes within each partially labeled public dataset. important to note that the BTCV [46] and AMOS22 [40] datasets already have original annotations for our eight target organs. By analyzing the curve depicting the decreasing value of the attention map, we observed that among the 5,595 CT volumes from the 16 partially labeled abdominal datasets, only
5

---

(8)
(0) pseudo error
pseudo ground
(c) ground truth error
pseudo label
ground
Patudo Label Endi
Touth
Unaaudabie Probiem
Adjusable Problem
Data Recolution Problem
Speottic Organ Probiem
missing cavity In the stomach
unclear boundary
missing anlarged pancreas and bladder
protocol inconsistency
Figure 5: (a) Error taxonomy. To minimize repetitive errors during human revision, we collate and analyze common mistakes that occur during the data made by either Al or human annotators. (b) Errors in pseudo labels. For example, the cavity in the stomach is missing; the enlarged pancreas tail caused by a pancreatic tumor is missing; the enlarged gall bladder is missing.
(c) Errors human be attributed to the ambiguity in defining organ
annotations. The high annotatoo variability or tumor boundaries, whereas the high inter-annote mnter-annotator variability often a result inconsistent annotation protocols and guidelines across different institutes.
5% of the volumes exhibited significantly large attention map values These high values indicate a higher risk of prediction errors, suggesting the need for further revision. Extrapolating this finding to a larger dataset of 8,000 CT volumes, we can estimate that the annotator will need to confirm and revise approximately 400 CT volumes. Assuming rate of 15 minutes per CT volume and 8-hour workday, this process would take approximately 12.5 days to complete. The attention map values are expected to decrease after fine-tuning Mo with revised labels as new benchmark.
Figure 5(a) summarizes typical errors encountered the active learning procedure into two categories: pseudo label errors and ground truth errors. As illustrated in Figure 5(b), pseudo label errors refer to the errors in the Al predictions. These errors often arise from irregular organ shapes, such the absence of cavity in the stomach the omission of enlarged pancreas and gall bladder. Discrepancies in CT volumes between the training and testing data, such as variations in scanners, protocols, reconstruction methods, and contrast enhancement, inaccuracies. These issues result in the model's predictions lacking accuracy in representing these specific anatomical structures. As exampled in Figure 5(c), ground truth errors are errors AI model predictions that result from inaccuracies in the human annotations used for training the model. These inaccuracies may arise due to unclear organ boundaries or inconsistency labeling protocols across different institutions, introducing variations into the model's predictions.
also contribute these
3.3 Dataset Construction
Annotators. Our study recruited three annotators, comprising a senior radiologist with over 15 years of experience and two junior radiologists with three years of experience. The senior radiol- ogist undertook the task annotation revision in the active learning procedure. Before releasing AbdomenAthe Atlas-KK, two junior radiologists looked through the masks in the entire AbdomenAtles 8Kand made revisions if needed (i.e., our method missed the error regions) In addition, the two junior radiologists conducted the inter-annotator variability analysis (Figure 8) and recorded the time for conventional methods when each organ must be annotated voxel by voxel (Appendix Table 4).
Efficiency. Why 30.8 years? We considered an 8-hour workday, five days a week. A trained annotator typically needs 60 minutes per organ per CT volume [65]. Our AbdomenAtlas-Se has total of eight organs and around 8,000 CT volumes. Therefore, annotating the entire dataset requires 60x 8 x 8000 (minutes) / 60/8/5 = 1600 (weeks) 30.8 (years). Why three weeks? Using our
'The 5% is empirically estimated based (1) the observation the distribution of the attention size (i.e., the number outliers in Figure and (2) annotation budget at each Step in active learning procedure. there are many outliers or limited budget, the threshold needs to increased accordingly.
Such revisions seldom required based on our study only 55 out of 8,448 volumes needing adjustments.

---

Table 1: Evaluation metries JHH. The sensitivity and precision evaluated between organ attention maps and organ error regions. Our attention map exhibits high average sensitivity and precision, indicating its effectiveness and accuracy in detecting false positives (FP) and false negatives (FN). This highlights its capability to precisely identify regions that need revision.
Metrics
Spl
RKid
LKid
Gall
Liv
Sensitivity
0.91 +: 0.25 0.68 0.40
0.93 1: 0.17
0.92 +: 0.18 0.85 0.23
0.99 1: 0.07
0.74 4: 0.33 0.88 0.12
Precision
0.90 +
0.67 =
Metrics
Stn
Aor
IVC
Pan
Avg.
Sensitivity
0.98 + 0.10
0.98 + 0.11
0.98 + 0.09
0.90 + 0.21
0.93 + 0.17
Precision
0.85 0.16
0.82 = 0.21
0.75 0.22
0.91 0.15
0.81 0.22
ground truth
pseudo label
false positives
false negatives
error region
attention map
Figure 6: Error region attention map. The discrepancy between the model's pseudo labels and the ground truth delineates the error regions in predictions, comprising both false positives (FP) and false negatives (FN). These error regions serve the benchmark for evaluating the sensitivity and precision of our attention map ($.
active learning strategy, only 400 CT volumes require manual revision from the human annotator (15 minutes per volume). That is, managed accelerate the annotation process by factor of 60x8/15=32 per CT volume. Therefore, we completed the entire annotation procedure within three weeks, as reported in the paper. Human efforts: x00x 15 (minutes) 60/8 12.5 (days) plus the time commitment for training and testing AI models taking approximately 8.5 (days).
split them into training (5500 CT volumes), validation (500), and test (2448) sets. Each volume contains vox-v annotations of the spleen, liver, left& right kidneys. stomach, gallbladder, pancreas, aorta, and IVC. Note that the JHH dataset is a proprietary, multi-resolution, multi-phase collected from Johns Hopkins Hospital [92, 42, 100, 47). This dataset used for external validation, including the assessment of the attention map quality (Table 1), Al generalizability (Table 2), the improvement of label quality in the active learning procedure (Appendix Table 5), and the performance of AI trained combination of public datasets and our AlalsmeA
Dataset splits. For 8, 8,448 CT volumes in AbdomenAtlas-8K,
Table 6).
Experiment & Result
4.1 Attention Map and Annotation Evaluation
Evaluation of attention map. Our attention map is evaluated 1,000 CT volumes of the JHH dataset [92], which is not included in the training process. The evaluation is performed across the eight target organs. Once error is detected, counts as a hit; otherwise, the quality of the attention map, two metrics are used: Sensitivity = TP / (TP + FN) and precision TP / (TP FP), where true positive (TP) means the attention map found real mistakes the AI made; a false negative (FN) means it missed some of the AI's mistakes; and a false positive (FP) means found mistakes where the AI was actually right. Sensitivity and precision are all calculated at the volume (a group of voxels) level rather than the voxel level. We chose sensitivity and precision because this experiment is designed evaluate an error detection task rather than segmentation task (comparing the boundary of attention maps and error regions). They can measure how well attention maps detect the real error regions and whether the errors being detected are real errors, respectively. The results of the attention map evaluation are presented in Table 1. The mean values of sensitivity and precision metrics for our attention maps, with respect to eight target organs, are 0.93+0.17 and 0.8 +0.22. Figure 6 shows the error regions and our attention map. These results
miss. To evaluate
7

---

ground truth
(a)
CT Scan
Pseudo Label
Revised Label
(b) CT Scan
Before
After
Figure 7: (a) Pseudo revised labels. Pseudo labels predicted by Al and then revised by annotators based on attention maps, resulting in revised labels. (b) AT predictions before vs. after fine-tuning, The post-fine-tuing results exhibit superior accuracy in segmenting the organ compared to the pre-fine-tuning result. Appendix Table quantifies the DSC scores before and after fine-tuning.
Swin UNETR
(IVC)
Smit UNETR
Swin UNETR
Spleen
Swin UHETR
Kidney
0.88
0.85
0.99
0.95
0.95
0.93
0.90
U-Met
net U-Net
U--oot
U-Net
0.80
0.93
0.95
0.95
0.80
0.90
and Net
nnti-Wet
nnu-Net
0.78
0.75
0.77
0.89 0.80 0.94
0.82
0.81
0.91
0.71 0.70
0.89
68
0.89
85
0.82 0.83
Swin UNETRR
U-fett
mott-Net
GT
Swin UNETR U-Wet
notl-Wiet
GT
Swin UNETR
L--ee
noU Not
Siein UNITRR U-WWr
not Net
Aorta
Stomach
Pancreas
Gallbladder
(a) architecture variability
Swin UNETRR
Kidney
Swin UNETR
Liver
Swin UNETR
1.0
0.93
0.97
0.96
0.8
Dr3
Dr1
Dr3
0r2
0.85
D22
0.94
0.90
Dr2
0.85
0.81
0.4
Swin UNETR
Dr1
Dr2
Swin UNETR
Dr1
Dr2
Swin UNETR
Dr1
D12
0.2
Gallbladder
Stomach
(b) Inter-annolator variability
Figure 8: The DSC score in each cell is calculated between the corresponding row and column. (a) AI architecture variability. The segmentation predictions made by different Al architectures display minor variations for eight organs, evidenced by the corresponding DSC scores. Overall, the Al predictions align closely with the annotations provided by human experts. (b) Inter-annotator variability. The DSC scores between AI predictions and human annotators consistently outperform the score between the two human annotators, suggesting a comparable annotation quality between AI predictions and human annotations in the segmentation of six organs,
suggest that our attention map effectively captures the false positives and false negatives in error regions, demonstrating high degree of accuracy identifying prediction errors.
Evaluation of human annotation quality. We provided the visualization of pseudo labels (generated by AI) and revised labels (annotated by humans) in Figure 7()). The revised labels were used to fine-tune the AI; we then compared the pseudo labels predicted by the AI before and after fine-tuning for the same CT volume from the JHH dataset, as shown Figure 7(b). After fine-tuning the Al using the revised labels, the AI was able to accurately segment the entire stomach on the previously unseen CT volumes. This demonstrated the high quality of the revised labels and their efficacy in enhancing Al performance. We further quantified the human annotation quality measured by Dice Similarity Coefficient (DSC) and normalized surface Dice (NSD) using 1,000 CT volumes from JHH dataset. The results in Appendix Table 5 show continual improvements in label quality along the active learning procedure. For example, Al models exhibited a marked improvement the (pseudo) labels of aorta and postcava, jumping from 72.3% to 83.7% and 76.1 16.1% % to 78.6%, respectively.

---

4.2 Label Bias and Segmentation Quality Evaluation
Evaluation label bias. We first evaluate the predictions made by three Al architectures, i.e., Swin UNETR, UnU-Net, and U-Net, on CT volumes derived from the JHH dataset [92]. The segmenta- tion predictions these architectures are illustrated Appendix Figure 10. We use the original annotations of the JHH dataset [92], which include eight target organs. Consequently, for each CT volume, we have four predictions: three from the AI models and one from human experts. We then compute the DSC score between each pair across cight organs. These results are presented in Figure 8(a) and demonstrate that while the three Al architectures produce slightly varied predictions, they closely resemble those performed by human experts. To prevent label bias to a specific archi- tecture, the final annotations in our AbdomenAtlas--8 are determined by averaging the three AI predictions. Our approach contrasts with the TotalSegmentator dataset [89], which exclusively relies on nnU-Net for segmentation. This could potentially generate biased dataset that impedes model generalization across different architectures scenarios.
Evaluation of AI prediction quality. To assess the automated annotation quality in AbtamenAtlas- 8K, we asked for the assistance of two additional human annotators to help modify the pscudo labels of our target eight organs. Due to time limitations, they are only able to revise six out of the eight organs, specifically on the 17 CT volumes from the BTCV dataset [46]. We compute DSC scores comparing Al predictions with those annotated independently by two human annotators, referred to Dr1 and Dr2. Figure 8(b) presents the results for six organs. The DSC scores between AI predictions and each human annotator (Swin UNETR Vs. Dr1 or Swin UNETR vs. Dr2) consistently exceed the score between the two human annotators (Drl vs. Dr2). These findings suggest that the automated Al annotations fall within the range of inter-annotator variability, thereby indicating that the quality of automated annotations comparable human annotations.
4.3 Benchmarking
AbdomenAtlas enables precision medicine for various downstream applications. We showcased one of the most pressing applications- carly detection and localization of pancreatic cancer, an extremely deadly disease with 5-year relative survival rate of only 12% in the United States. The AI trained on a large, private dataset at Johns Hopkins Hospital (JHH) performed arguably higher than typical radiologists [92, 47, 42]. But this Al model and annotated dataset were inaccessible due to the many policies. Now, our paper demonstrated that using AbdomenAtlas of publicly accessible CT volumes), AI can achieve similar performance when directly tested on the can
(100% made up
JHH dataset (see Table 2). This study is a concrete demonstration of how AbdomenAtlass be used to train AI models that can be generalized many CT volumes from novel hospitals and be adapted to address a range of clinical problems. For a more in-depth analysis of the segmentation performance of AI models, we present the comprehensive category- -ise scores comparison across eight organs in Appendix Table 7.
Benchmark results of AI models trained on AblomenA We directly apply four AI models, i.e. SwinUNETR, UNETR, U-Net, and SegResNet, trained on AbdomenAtlas-e (public) to JHH (unseen, private) and compared its performance with Al trained JHH. The AI models trained on AbdomenAtlas--8 exhibit comparable results in average mDSC and mNSD across eight organs when compared with Al models trained on the JHH dataset, proving the high generalization capacity of AI models trained on AbdomenAtlas-83
Table 2:
Trained on JHH (private)
Trained nn AbdomenAtias--B (public) mNSD (%)
AI Models
mDSC (%)
mNSD (%)
mDSC (%)
SwinUNETR [84]
84.8 = 12.6
66.5 14.8
86.5
60.8 10.9
UNETR [28]
78.6 : 13.0
53.9 14.1
86.6 1: 6.5
59.5 1 10.4
U-Net [73]
84.8 11.2
64.2 1: 14.
87.3 6.0
61.0 + 10.2
SegResNet [12]
6.60
87.0
60.4 10.1
5
Discussion
Impact. The scientific community has generally agreed that large volumes of annotated data are required for developing effective AI algorithms [39, 2, 60, 24, 27, 101]. For example, developing

---

Foundation Models for healthcare has recently raised much attention. The Foundation Model refers to an Al model that is trained on a large dataset and can be adapted to many specific downstream applications. This requires a large-scale, fully-annotated dataset. The currently available medical datasets are too small to represent the real data distribution in clinics [80, 13, 68). The availability of large-scale, multi-center, fully-labeled data (summarized in Appendix Table 3) is one of the most significant cornerstones for both the development and evaluation of CADe systems [90, 103]. In recent years, the rise of imaging data archives [74, 57, 38, 3, 14, 16, 58, 97, 49, 99] and international competitions [45, 86, 78, 79, 30, 29, 1, 15, 4, 40] produced several publicly available datasets for benchmarking AI algorithms, but these datasets usually are of small size, contain partial labels, come from various scanners & protocols, and are therefore often limited in their scope [22, 50, 102, 95, 96, 77, 67. 88, 42]. We anticipate that our AadommnnAlla--ae the model to capture complex organ patterns, variations, and features across different imaging phases, modalities, and wide range of populations. This has been partially evidenced in Appendix Table 6, wherein we generalize Al to CT volumes taken from different hospitals.
can play an important role in enabling
Limitation. Pscudo-labels have the potential to expedite tumor annotation procedures as well, but the risk of producing a large number of false positives is a significant concern. The presence of false positives could significantly increase the time annotators need to accept or reject a detection. In total, our Al models generate 51,852 tumor masks, including tumors in the colon, liver, hepatic vessel, pancreas, kidneys, and lung. However, only using AbdsmenAtlas-SK, we not able to assess the Al performance of tumor detection due to the lack of comprehensive pathology reports or expert annotations to describe the tumors in the majority of these publicly available CT volumes. To address this problem, we collected 392 tumor-free (private) CT volumes from Johns Hopkins Hospital to evaluate the false positives in the pseudo labels. Using kidney tumor detection as an example, there 37 out of 392 CT volumes containing false positives (FPR = 9.4%) and a total of 161 false positives in the 37 CT volumes. Furthermore, we stratified the dataset based on the type of blood vessels, identifying average false positive rates of 11.83% and 4.6% for the venous and arterial vasculature, respectively. Therefore, while pseudo labels can be helpful, we anticipate more effective method for false positive reduction is needed to enable clinically practical tumor annotation, given the complexity of tumors compared with organs. As an extension, we plan add tumor annotations to AbdomenAAils-Sk in three possible directions. Firstly, we plan to recruit more experienced radiologists to revise tumor annotations. Secondly, we will incorporate the pathology reports (based on biopsy results) into the human revision. These actions can reduce potential label biases and label errors from human annotators. Thirdly, we will exploit synthetic data (tumors) that can produce enormous tumor examples and their precise masks for AI training and validation [32, 47,33].
In TotalSegmentator, the labels were largely generated by single nnU-Net re-trained continually. Depending solely on nnU-Net could introduce a potential label bias favoring the nnU-Net architecture. This means that whenever TotalSegmentator is employed for benchmarking, unU-Net would always outperform other segmentation architectures (e.g., UNETR, TransUNet. Swin UNETR, etc.). This observation has been made in several publications such as Huang al. [35]. In contrast, our AbdomenAtlla-SK incorporates predictions from three different Al architectures, preventing bias towards one specific architecture [89]. However, such a solution comes with increased computational costs, and the performance of the Al architectures can vary. Taking an average of the predictions may result in the final outcome being pulled down by poorly performing Al architectures.
Conclusion
Our study shows the effectiveness of an active learning procedure, which combines the expertise of an annotator with the capabilities of a trained AI model. It not only deploys multiple AI models to detect prediction errors but also prompts annotators to revise these potential failures. Using this approach, we successfully annotated 8,448 abdominal CT volumes of different organs within three weeks, expediting the annotation process by staggering factor of 533. Furthermore, our experiments demonstrate that the interaction between human and AI models can produce comparable or even superior results than that of human annotator alone. This indicates that our approach can leverage the strengths of both human and AI models to achieve accurate and efficient annotation. Leveraging our efficient annotation framework, we anticipate that larger-scale medical datasets of various modalities, organs, and abnormalities can be curated at significantly reduced cost, ultimately contributing to the development of Foundation Models in the medical domain [91, 52, 61].
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the Patrick J. McGovern Foundation Award. We appreciate the effort of the MONAI Team to provide open-source code for the community. This work has partially utilized the GPUs provided by ASU Research Computing. We thank Elliot K. Fishman, Linda Chu, and Satomi Kawamoto for providing the JHH dataset for external validation; Yuxiang Lai for generating the 3D rendering of segmentation; thank Xiaoxi Chen for reviewing and revising AI predictions; thank Seth Zonies and Andrew Wichmann for providing legal advice on the release of Abdomen A Atlas-8K; thank Yu-Cheng Chou, Jieneng Chen, Junfei Xiao, Wenxuan Li, and Xiaoding Yuan for their constructive suggestions at several stages of the project. The content and dataset of this paper are covered by patents pending.
References
[1] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Bennett A Landman, Geert Litjens, Bjoemn Menze, Olaf Ronneberger, Ronald Summers, Bram van Ginneken, et The medical segmentation decathlon, arXiv preprini wxXiv22106.07335 2021.
[2] Diego Ardila, Atilla P Kiraly, Sujeeth Bharadwaj. Bokyung Choi, Joshua J Reicher, Lily Peng. Daniel Tse, Mozziyar Etemadi, Wenxing Ye, Greg Corrado, et al. End-to-end lung cancer screening with three- dimensional deep learning on low-dose chest computed tomography. Nature medicine, 25(6):954 961, 2019.
[3] Samuel G Armato III. Geoffrey McLennan, Lue Bidaut, Michael F McNitt- Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lide) and image database resource initiative (idri): a completed reference database of lung nodules on et scans. Medical physics, 38(2)9155995) 1-1911, 2011.
[4] Ujjwal Baid, Satyam Ghodasara, Michel Bilello, Suyash Mohan, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Holpathy-Cramee Felipe Kitamura, Sarthak Pati, al. The rsna-asnr-micee miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107. 107.02314, 2021.
[5] Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin hased active learning. In International Conference on Computational Learning Theory, pages 35-50. Springer, 2007.
[6] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, al. The liver tumor segmentation benchmark (lits). Medical Image Analyxis, 84: 80:600, 2023.
[7] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi- Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (lits), arXiv preprint wxXiv:1901.0405: 2019.
[8] Samuel Budd, Emma C Robinson, and Bernhard Kainz, A survey on active learning and human-in-the. loop deep learning for medical image analysis. Medical Image Analysis, page 102062, 2021.
[9] Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797, 2020.
[10] Jieneng Chen, Yingda Xia, Jiawen Yao, Ke Yan, Jianpeng Zhang, Le Lu, Fakai Wang, Bo Zhou, Mingyan Qiu, Qthang Yu, et al. Towards a single unified model for effective detection, segmentation, and diagnosis of eight major cancers using large collection of scans. arXiv preprint wXXv:2201.112911 2023.
[11] Liangyu Chen, Yutong Bai. Siyu Huang, Yongyi Lu, Bihan Wen, Alan L Yuille, and Zongwei Zhou. Making your first choice: To address cold start problem in vision active learning. In Medical Imaging with Deep Learning. 2023.
[12] Xinze Chen, Guangliang Cheng. Yinghao Cai, Dayong Wen, and Heping Li. Semantic segmentation with modified deep residual networks. In Pattern Recognition: 7th Chinese Conference, CCPR 2016, Chengdu, China, November 5-7, 2016, Proceedings, Part 7, pages 42 54. Springer, 2016.
[13] Linda C Chu and Elliot K Fishman. Deep learning for pancreatic cancer detection: current challenges and future strategies. The Lancet Digital Health, 26))e271-2272 2020.
11

---

[14] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel. Stephen Moore. Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating public information repository. Journal of digital imaging, 26(6):1045-1055 2013.
[15] Errol Colak, Felipe C Kitamura, Stephen Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy- Rohyn Ball, George Shih, Anouk Stein, et al. The rsna pulmonary embolism et dataset. Radiology: Artificial Intelligence, 3(2)xe200254, 2021.
[16] Karen L Crawford, Scott Neu, and Arthur W Toga. The image and data archive at the laboratory of neuro imaging. Neuroimage, 124:1080-188 1003, 2016.
[17] A Criminisi. Microsoft research cambridge (msrr) object recognition image database (version 2.0), 2004.
[18] Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pages 746-751, 2005.
[19] Ido Dagan and Sean P Engelson. Committee- based sampling for training probabilistic classifiers. In Muchine Learning Proceedings 1995, pages 150-157. Elsevier, 1995.
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248 255. IEEE, 2009.
[21] Yang Deng, Ce Wang, Yuan Hui, Qian Li, Jun Li, Shiwei Luo, Mengke Sun, Quan Quan, Shuxin Yang. You Hao, et al. Ctspinelk: large-scale dataset for spinal vertebrac segmentation in computed tomography. arXiv preprint arXiv:2105. 14711, 2021.
[22] Nanqing Dong, Michael Kampffmmyer, Xiaodan Liang, Min Xu, Irina Voiculescu, and Eric Xing. Towards robust medical image segmentation on small-scale data with incomplete labels. arXiv preprint arXiv:201 J. /4164,1444444 2020.
[23] Matthias Eisenmann, Annika Reinke, Vivienn Weru, Minu Dietlinde Tizabi, Fabian Isensee, Tim J Adler, Sharib Ali, Vincent Andrearczyk, Marc Aubreville, Ujjwal Baid, et al. Why is the winner the best? arXiv preprint 23.3::2303.17119 2023.
[24] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Se- bastian Thrun Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):1155 15, 2017.
[25] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1193- PMIR, 2017.
[26] Eli Gibson, Francesco Giganti, Yipeng Hu, Ester Bonmati, Steve Bandula. Kurinchi Gurusamy, Brian Davidson, Stephen P Percira, Matthew Clarkson, and Dean C Barratt. Automatic multi-organ segmenta- tion on abdominal ct with dense v-networks. IEEE transactions on medical imaging. 37(8):8822- -8244 2018.
[27] Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation a deep learning algorithm for detection diabetic retinopathy in retinal fundus photographs. Jama, 316(22):2402- 2410, 2016.
[28] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574-584 774-544, 2022.
[29] Nicholas Heller, Fabian Isensce, Klaus H Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, et al. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits challenge. Medical image analysis, 67:101821, 2021.
[30] Nicholas Heller, Sean MeSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Ocstreich, Paul Blake, Joel Rosenberg, et al. An international challenge to use artificial intelligence to define the state-of-thc-art in kidney and kidney tumor segmentation in ct imaging.. 2020.
[31] Hideitsu Hino. arXiv:2012.04225, 2020.
Active learning: Problem settings and recent developments. arXiv preprint
12

---

[32] Qixin Hu, Yixiong Chen, Junfei Xiao, Shuwen Sun. Jieneng Chen, Alan L Yuille, and Zongwei Zhou. Label-free liver tumor segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7422-7432, 2023.
[33] Qixin Hu, Junfei Xiao, Yixiong Chen, Shuwen Sun, Jie-Neng Chen, Alan Yuille, and Zongwei Zhou. Synthetie tumors make ai segment tumors hetter. NeurIPS Workshop on Medical Imaging meets NeurlPS, 2022.
[34] Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al. Segment anything model for medical images? arXiv preprint arXiv:2304. 14660, 2023.
[35] Ziyan Huang, Haoyu Wang. Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He. Yun Gu, Lixu Gu. Shaoting Zhang, et al. Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training. arXiv preprint Ttt:2304.067/66 2023.
[36] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund. Behzad Haghgoo. Robyn Ball, Katie Shpanskaya, al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence volume 33, pages 590-597, 2019.
[37] Fabian Isensee, Paul Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein, nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203-211, 2021.
[38] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L.. Whitwell, Chadwick Ward, al. The alzheimer's disease neuroinaging initiative (admí): Mri methods. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, 27(4))685-691, 2008
[39] Saurabh Jha and Eric J Topol. Adapting to artificial intelligence: radiologists and pathologists as information specialists. Jama, 316(22);2353-23544 2016.
[40] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206. 08023, 2022.
[41] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng. Roger G Mark, and Steven Horng. Minic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):188, 2019.
[42] Mintong Kang, Bowen Li, Zengle Zhu, Yongyi Lu, Elliot K Fishman, Alan Yuille, and Zongwei Zhou. Label-assemble: Leveraging multiple datasets with partial labels. In IEEE 20th International Symposium on Biomedical Imaging (ISBI). IEEE, 2023.
[43] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yon Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[44] Johannes Kulick, Robert Lieck. Mare Toussaint, et al. Active learning of hyperparameters: An expected cross entropy criterion for active model selection. ArXiv e-prints, 2014.
[45] Bennett Landman, Zhoubing Xu, Juan Eugenio Igelsias, Martin Styner, Thomas Robin Langerak, and Arno Klein. Multi-atlas labeling beyond the cranial vault-workshop and challenge. 2017.
[46] Bennett Landman, Zhoubing Xu, Igelsias, Martin Styner, T Langerak, and Amo Klein. Miccai multi- atlas labeling beyond the cranial vault- worksho and challenge. In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Works-Workshop Challenge, volume 5, page 12, 2015.
[47] Bowen Li, Yu-Cheng Chou, Shuwen Sun, Hualin Qiao, Alan Yuille, and Zongwei Zhou. Early detection and localization of pancreatic cancer by label-free tumor synthesis. MICCAI Workshop on Big Task Small Data, 1001-AI, 2023.
[48] Xin Li and Yuhong Guo. Adaptive active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 859-866, 2013.
[49] Yuexiang Li and Linlin Shen. Skin lesion analysis towards melanoma detection using deep learning network. Sensors. 18(2)5556, 2018
13

---

[50] Pengbo Liu, Li Xiao, and S Kevin Zhou. Incremental learning for multi-organ segmentation with partially labeled datasets. arXiv preprint arXiv:2103.05526 2021.
[51] Xiangde Luo, Wenjun I Liao, Jianghong Xiao, Tao Song, Xiaofan Zhang, Kang Li, Guotai Wang, and Shaoting Zhang. Word: Revisiting organs segmentation in the whole abdominal region. arXiv preprint arXiv:2111.02433. 2021.
[52] Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304. 123306, 2023.
[53] Jun Ma, Yao Zhang, Song Gu, Xingle An, Zhihe Wang, Cheng Ge, Congcong Wang, Fan Zhang, Yu Wang, Yinan Xu, et al. Fast and low-gpu-memory abdomen ct organ segmentation: the flare challenge. Medical Image Analysis, 82: 52:102616, 2022.
[54] Jun Ma, Yao Zhang, Song Gu. Cheng Ge. Shihao Ma, Adamo Young, Cheng Zhu, Kangkang Meng, Xin Yang. Ziyan Huang, et al. Unleashing the strengths unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge. arXiv preprint arXiv:2308.08. 05862, 2023.
[55] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al. Abdomenct- Ik: Is abdominal organ segmentation a solved problem. IEEE Transactions on Pattern Anaysss and Machine Intelligence, 2021.
[56] Dwarikanath Mahapatra, Behzad Bozorgtahar, lean-Philippe Thiran, and Mauricio Reyes. Efficient active learning for image classification and segmentation using a sample selection and conditional generative adversarial network. In International Conference on Medical Image Computing and Compuder-Assine Intervention, pages 580-588. Springer, 2018
[57] Daniel S Marcus, Tracy Wang. Jamie Parker, John Csemannkky, John Morris, and Randy Buckner. Open access series of imaging studies (oasis): cross-sections mri data in young. middle aged, nondemented, and demented older adults. Journal cognitive neuroscience, ((9)):498-1555 2007.
[58] Mojtaba Masoudi, Hamid Reza Pourreza, Mahdi Saadatmand Tarzjan, Noushin Eftekhari, Fateme Shafiee Zargar, and Masoud Pezeshki Rad. A new dataset of tomourrppyy angiography images for computer-aided detection of pulmonary embolism. Scientific data, 5(1):1 (00)119,220 2018.
[59] Andrew Kachites McCallumzy and Kamal Nigamy. Employing em and pool-based active learning for text classification. In Proc. International Conference on Machine Learning, pages 359 367.. Citeseer, 1998.
[60] Scott Mayer McKinney, Marcin Sieniek. Varun Godbole, Jonathan Godwin. Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado, Ara Darzi, et al. International evaluation of an ai system for breast cancer screening. Nature, 577(7788):89 0111899994,499 2020.
[61] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric Topol, and Pranay Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 61 616(7956):259-28 22659926662220 2023.
[62] Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, and Shadab Khan. Towards robust and reproducible active learning using neural networks. ArXiv, abs/2002.0056// 2020.
[63] Vishwesh Nath, Dong Yang, Holger R Roth, and Daguang Xu. Warm start active learning with proxy labels and selection semi-super fine-tunin fine-tuning International Conference on Medical Image Computing and Computer-Assisee Assistee Intervention, pages 297-308. Springer, 2022.
[64] Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision, 42(3):145 175, 2001.
[65] S Park, LC Chu, EK Fishman, AL Yuille, B Vogelstein, KW Kinzler, KM Horton, RH Hruban, ES Zinreich, D Fadaci Fouladi, al. Annotated normal et data of the abdomen for deep learning: Challenges and strategies for implementation. Diagnostic and interventional imaging, 101(1):35 44, 2020.
[66] Amin Parvaneh, Ehsan Abbasnejad. Damien Teney, Gholamreza Reza Haffari, Anton Van Den Hengel, and Javen Qinfeng Shi. Active learning by feature mixing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 12237-12246, 2022.
[67] Olivier Petit, Nicolas Thome, and Luc Soler. Iterative confidence relabeling with deep convnets for organ segmentation with partial labels. Compaterized Medical Imaging and Graphics, page 101938, 2021.
[68] Deborah Plana. Dennis L Shung, Alyssa A Grimshaw, Anurag Saraf, Joseph JY Sung, and Benjamin H Kann. Randomized clinical trials of machine learning interventions in health care: a systematic review. e2233946 2022.
JAMA Network Open, 249992222344--223994
14

---

[69] Luciano M Prevedello, Safwan S Halabi, George Shih, Carol C Wu, Mare D Kohli, Falgun II Chokshi. Bradley Erickson, Jayashree Celpathy-Crammes Katherine P Andriole, and Adam E Flanders. Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiology: Artificial Intelligence, 1(1)e188031, 2019.
[70] Pranav Rajpurkar and Matthew P Lungren. The current and future state of ai interpretation of medical images. New England Journal of Medicine, 388(21):1981 199111999199999111111111911999 2023.
[71] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. survey of deep active learning. arXiv preprint 20Xx:20009002266 2020.
[72] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org, a new dataset for multiple organ segmentation in computed tomography. Scientific Duta, 1(1):199, 2020.
[73] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 234-241. Springer, 2015.
[74] Holger R Roth. Le Lu, Amal Farag, Hoo-Chang Shin. Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation. International conference on medical image computing and conputer-assisted intervention, pages 556-564, Springer, 2015.
[75] Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International Symposium on Intelligent Data Analysis, pages 309-318. Springer, 2001.
[76] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1708.00489, 2017.
[77] Gonglei Shi, Li Xiao, Yang Chen, and S Kevin Zhou. Marginal loss and exclusion loss for partially supervised multi-organ segmentation. Medical Image Analysis, 70: 10:101979, 2021.
[78] George Shih, Carol C Wu, Safwan Halabi, Mare D Kohli, Luciano M Prevedello, Tessa Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya
al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiology: Artificial Intelligence, 1(1):e 180041, 2019.
[79] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Gin- neken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoemn Menze, al. large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv: 000.999920000.3 2019.
[80] Shelly Soffer, Avi Ben-Cohen, Orit Shimon, Michal Marianne Amitai, Hayit Greenspan, and Eyal Klang. Convolutional neural networks for radiologic images: radiologist's guide. Radiology, 290(3)55006666, 2019.
[81] Jamshid Sourati, Ali Gholipour, Jennifer G Dy, Sila Kurugol, and Simon K Warfield. Active deep learning with fisher information for patch-wise semantic segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 83-91. Springer, 2018
[82] Jamshid Sourati, Ali Gholipour, Jennifer G Dy, Xavier Tomas- Fernandez, Sila Kurugol, and Simon Warfield. Intelligent labeling based on fisher information for medical image segmentation using deep learning. IEEE transactions on medical imaging, 38(1 1):2642 226222222633222 2019.
[83] Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey N Chiang, Zhihao Wu, and Xiaowei Ding. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis, page 101693, 2020.
[84] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 20730- 20740, 2022.
[85] Emily B Tsai, Scott Simpson, Matthew P Lungren, Michelle Hershman, Leonid Roshkovan, Errol Colak, Bradley J Erickson, George Shih, Anouk Stein. Jayashree Kalpathy-Cramer, covid-19 19 open radiology database (ricord). Radiology, 19204) E213, 2021.
Cramer, et al. The rsna international
15

---

[86] Vanya V Valindria, Nick Pawlowski, Martin Rajchl, Ioannis Lavdas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Multi-modal learning from unpaired images: Application multi- organ segmentation in et and mri. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 547-556. IEEE, 2018.
[87] Jingwen Wang. Yuguang Yan, Yubing Zhang, Guiping Cao, Ming Yang. and Michael K Ng. Deep reinforcement active learning for medical image classification, In Medical Image Computing and Computer Assisted Intervention-MICCA/ 2020: 23nd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part 23, pages 33-42. Springer, 2020.
[88] Li Wang. Dong Li, Yousong Zhu, Lu Tian, and Yi Shan. Cross-dataset collaborative learning for semantic segmentation. arXiv preprint arXiv:2103. 2021.
[89] Jakoh Wasserthal, Manfred Meyer, Haans-CCristiian Breit, Joshy Cyriac, Shan Yang. and Martin Segeroth. Totalsegmentator:: rohust segmentation of 104 anatomical structures in et images. arXiv preprint arXiv:2208 05868, 2022.
[90] Martin J Willemink, Wojciech A Koszek, Cailin Hardell, Jie Wu, Dominik Fleischmann, Hugh Harvey. Les R Folio, Ronald M Summers, Daniel L Rubin, and Matthew P Lungren. Preparing medical imaging data for machine learning. Radiology, 295(1):4-15, 2020.
[91] Martin Willemink, Holger R Roth, and Veit Sandfort. Toward foundational deep learning models for medical imaging in the new era of transformer networks. Radiology: Artificial Intelligence, 4(6):211224, 2022.
[92] Yingda Xia, Qihang Yu, Linda Chu, Satomi Kawamoto, Seyoun Park, Fengze Liu, Jieneng Chen, Zhuotun Zhu, Bowen Li, Zongwei Zhou, et al. The felix project: Deep networks detect pancreatic neoplasms. medRxiv, 2022.
[93] Yingda Xia, Qihang Yu, Linda Chu, Satomi Kawamoto, Seyoun Park, Fengze Liu, Jieneng Chen, Zhuotun Zhu, Bowen Li, Zongwei Zhou, Yongyi Lu, Yan Wang, Wei Shen, Lingxi Xic, Yuyin Zhou, Daniel Fouladi, Shahab Shayestch, Scott Jefferson Graves, Alejandra Blanco, Eva Zinreich, Ken Kinzler, Ralph Gruban, Bert Vogelstein, Eliot Fishman, and Alan Yuille. Ai algorithms can assist radiologists in early detection of pancreatic neoplasms through venous and arterial et imaging. In Radiological Society of North America (RSNA), 2022.
[94] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8068- 8078, 2022.
[95] Ke Yan, Jinzheng Cai, Adam P Harrison, Dakai Jin, Jing Xiao, and Le La. Universal lesion detection by learning from multiple heterogencously labeled datasets. arXiv preprint arXiv:2005. 13753, 2020.
[96] Ke Yan, Jinzheng Cai, Youjing Zheng, Adam P Harrison. Dakai Jin, You-bao Tang. Yu-Xing Tang. Lingyun Huang, Jing Xiao, and Le Lu. Learning from multiple datasets with heterogencous and partial labels for universal lesion detection in ct. IEEE Transactions on Medical Imaging, 2020.
[97] Ke Yan, Xiaosong Wang, Le Lu, and Ronald M Summers. Deeplesion: automated mining of large- scale lesion annotations and universal lesion detection with deep learning. Journal of medical imaging. 5(3):036501, 2018
[98] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In International conference on medical image compuling and computer-assistee intervention, pages 399-407. Springer, 2017.
[99] Yang Yang, Xueyan Mei, Philip Robson, Brett Marinelli, Minggian Huang, Amish Doshi, Adam Jacobi, Katherine Link, Thomas Yang, Chendi Cao, Ying Wang, Hayit Greenspan, Timothy Deyer, and Zahi Fayad. Radimagenet: A large-scale radiologic dataset for enhancing deep learning transfer learning research, 2021.
[100] Yuan Yao. Fengze Liu, Zongwei Zhou, Yan Wang, Wei Shen, Alan Yuille, and Yongyi Lu. Unsu- pervised domain adaptation through shape modeling for medical image segmentation. arXiv preprint arXiv:2207.00 02559, 2022.
[101] Alan L Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? International Journal of Computer Vision, 129(3):81-8802 2021.
16

---

[102] Jianpeng Zhang, Yutong Xie, Yong Xia, and Chunhua Shen. Dodnet: Learning to segment multi-organ and tumors from multiple partially labeled datasets. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1195-1204, 2021.
[103] Zongwei Zhou. Towards Ionotation-Efficiien Deep Learning for Computer-Aided Diagnosis. PhD thesis, Arizona State University, 2021.
[104] Zongwei Zhou, Michael B Gotway, and Jianming Liang. Interpreting medical images. In Intelligent Systems Medicine and Health, pages 343 371. Springer, 2022.
[105] Zongwei Zhou, Jae Shin, Ruibin Feng, R Todd Hurst, Christopher B Kendall, and Jianming Liang. Integrating active learning and transfer learning for carotid intima-media thickness video interpretation. Journal of digital imaging, 32(2):200-299 2019.
[106] Zongwei Zhou, Jae Shin, Zhang. Suryakanth Gurudu, Michael Gotway, and Jianming Liang. Fine- tuning convolutional neural networks for biomedical image analysis: actively and incrementally. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7340-733 7351, 2017.
[107] Zongwei Zhou, Jae Y Shin, Suryakanth Gurudu, Michael Gotway, and Jianming Liang. Active, continual fine tuning of convolutional neural networks for reducing annotation efforts. Medical image analysis, 71:101997, 2021.
17


Reviews:
[{"review_id": "CbGFR0sgWD", "reviewer": "Reviewer_3MZ3", "summary": "", "questions": "", "limitations": "YES", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. **Simple but Effective Idea:** The submission uses multiple pre-trained models to reduce label bias in medical imaging annotation, demonstrating an innovative yet straightforward solution.\n\n2. **Clear Writing:** The paper is well-structured and articulates complex ideas clearly, making it accessible to a broad audience.\n\n3. **Large-Scale Dataset:** The creation of the largest annotated multi-organ dataset to date, with 8,448 CT volumes, is a major accomplishment that can greatly benefit the medical imaging field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "NBpNrP2pTl", "reviewer": "Reviewer_Tr7Z", "summary": "", "questions": "", "limitations": "The authors correctly point out that one of the limitations of the approach is evaluating the proposed methodology with respect to disease (e.g., when tumors are present). It would also be important to highlight that the algorithm used in the active learning step is supervised, therefore, any label biases from labels in the training datasets may potentially be propagated into the labelling of the new data.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well written and easy to follow. Method is used to create a large-scale segmentation dataset of eight organs over a large number of CT scans. Since annotating medical images and release of data is typically a large burden which prevents AI model development, such a dataset will be very useful to facilitate development and training of AI for medical applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2p4LWVtTEk", "reviewer": "Reviewer_pW4Y", "summary": "", "questions": "", "limitations": "-The reduction of abdominal labels to 8 is a limitation. Many smaller anatomical structures (adrenal glands, etc) are also of clinical interest and in comparison to the TotalSegmentator which provides 104 anatomies this is actually a step backwards.   \n-It remains somewhat unclear whether all the data will be publicly available upon acceptance. Since for the largest subset of the Abdomen8k JHH is is noted that no licence is available. \n-An important aspect of the active labelling procedure (5) is not described, the sentence is incomplete. \n-The comparison of training with this new dataset compared to previous partially labelled ones in Appendix A does not show a higher Dice 90.3 vs 90.4% and no statistical tests demonstrate the usefulness of this expanded dataset.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "-The strategy of combining complementary strengths of multiple segmentation models trained on the same limited dataset is compelling. \n-The proposed attention map for interactive segmentation refinement of human raters is useful and substantially reduces annotation time compared to a complete manual annotation.\n-The visualisations are thoroughout very well done and informative.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9NzEcwRUbk", "reviewer": "Reviewer_nhxF", "summary": "", "questions": "", "limitations": "No obvious limitations were identified in the body of work performed. The potential of labeling tumor in tissues in the future to reduce false positive labels, has already been discussed in the manuscript.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The construction of the large dataset and the use of active learning for annotation are highlights of the study. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "iuYdVR7lkR", "reviewer": "Reviewer_R6jP", "summary": "", "questions": "", "limitations": "A discussion on limitations is provided in the Discussion section.\n\nThe generalisability of the approach to other organs and imaging modalities is only mentioned briefly at the end of the conclusion. Some further discussion on the feasibility of this and the work that would be involved to adapt the software would be useful.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed algorithm is shown to provide rapid annotation possibilities.\n\nThe accuracy of the AI approach is compared favourably to annotations by two medical professionals.\n\nThe code is made available.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors have provided detailed responses addressing the reviewers' valuable feedback. I appreciate the authors' sincere efforts to improve the work based on the reviewers' suggestions.

A key concern was around the dataset release. The authors are working to enable the release of the private dataset of CT volumes. The code to generate the dataset using the proposed active learning approach has already been released.

Another question was around the generalizability of the approach to more structures and modalities. The authors have helpfully discussed explorations on expanding to 13 more organs, various tumors, and modalities like MRI. Integrating the approach into open-source tools will also aid adoption.

The comparisons between AbdomenAtlas-8K and TotalSegmentator were discussed further. The authors agreed that reducing annotations to 8 organs is a limitation but one stemming from their focus on active learning. Expanding annotations is planned. The time savings arise from prioritizing volumes needing revision versus reviewing all volumes.

Some metrics like specificity were added to address a reviewer's concern. Inter-annotator variability was also clarified - it assessed the AI model's quality rather than human bias. Overall, the authors' detailed responses and additions have satisfactorily addressed the reviewers' comments.

I recommend accepting this paper given the novelty of the large-scale annotated dataset achieved via an efficient active learning approach. This can enable training robust AI models for medical image analysis.
Decision: Accept (Poster)