SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks
Chongyu Qu $^{1}$  Tiezheng Zhang $^{1}$  Hualin Qiao $^{2}$  Jie Liu $^{3}$  Yucheng Tang $^{4}$  Alan L. Yuille $^{1}$  Zongwei Zhou $^{1,*}$
<sup>1</sup>Johns Hopkins University <sup>2</sup>Rutgers University <sup>3</sup>City University of Hong Kong <sup>4</sup>NVIDIA
Code & Data: https://github.com/MrGiovanni/AbdomenAtlas
Annotating medical images, particularly for organ segmentation, is laborious and time-consuming. For example, annotating an abdominal organ requires an estimated rate of 30–60 minutes per CT volume based on the expertise of an annotator and the size, visibility, and complexity of the organ. Therefore, publicly available datasets for multi-organ segmentation are often limited in data size and organ diversity. This paper proposes an active learning procedure to expedite the annotation process for organ segmentation and creates the largest multi-organ dataset (by far) with the spleen, liver, kidneys, stomach, gallbladder, pancreas, aorta, and IVC annotated in 8,448 CT volumes, equating to 3.2 million slices. The conventional annotation methods would take an experienced annotator up to 1,600 weeks (or roughly 30.8 years) to complete this task. In contrast, our annotation procedure has accomplished this task in three weeks (based on an 8-hour workday, five days a week) while maintaining a similar or even better annotation quality. This achievement is attributed to three unique properties of our method: (1) label bias reduction using multiple pre-trained segmentation models, (2) effective error detection in the model predictions, and (3) attention guidance for annotators to make corrections on the most salient errors. Furthermore, we summarize the taxonomy of common errors made by AI algorithms and annotators. This allows for continuous improvement of AI and annotations, significantly reducing the annotation costs required to create large-scale datasets for a wider variety of medical imaging tasks.
Medical segmentation is a rapidly advancing task that plays a vital role in diagnosing, treating, and radiotherapy planning [37, 10, 32, 103, 40]. Building datasets of a substantial number of annotated medical images is critical for training and testing artificial intelligence (AI) models<sup>1</sup> [104, 52, 70]. However, medical datasets carefully annotated by an annotator are infeasible to create at a huge scale using conventional annotation methods [65] because performing per-voxel annotations is expensive and time-consuming [89, 55, 53]. As a result, publicly available datasets for multi-organ segmentation are often limited in data size (a few hundred) and organ diversity<sup>2</sup> as reviewed in Figure 1(a).
*Corresponding author: Zongwei Zhou (ZZHOU82@JH.EDU)
<sup>1</sup>While some of the abdominal organs can be automatically segmented by state-of-the-art AI models with fairly high accuracy, many other organs and anatomical structures remain suboptimal or unknown [55].
${}^{2}$  TotalSegmentator [89] annotated a diverse set of classes for 1,024 CT volumes, but a majority of volumes were largely annotated only by pre-trained nnU-Nets [37]. As a result, their annotations are highly biased to the specific architecture (see §5); also, their procedure does not consider inconsistency across different AI models.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.
(b)
Figure 1: (a) An overview of public datasets. AbdomenAtlas-8K stands out from other datasets due to its large number of annotated CT volumes. We have reviewed dataset names (and licenses). (b) Volume distribution of eight organs. The significant variations within and across organs presented in our AbdomenAtlas-8K present challenges for the multi-organ segmentation and the generalizability of models to different domains. More comparisons can be found in Appendix Table 3 and Figure 9.
There is a pressing need to expedite the annotating procedure. The latest endeavors to construct a fully annotated dataset remained to ask radiologists to manually annotate each and every missing label [58, 36, 41, 78, 72, 9, 85, 21, 55, 99]—the same annotating procedure used a decade ago [20] or even earlier [64, 17]. Such a procedure is extremely costly, particularly for medical images, and will be time-consuming to create a large-scale dataset for every medical specialty [69, 90, 13, 83, 65]. To improve the annotation efficiency, active learning has been widely explored by combining the radiologist's competence and the computer's capability [106, 107, 71, 8, 11]. However, most studies in the active learning literature have been retrospective in nature, in which the annotating procedure was simulated by simply retrieving labels for the data without physically involving the radiologists in the loop to create/revise the labels [105, 66, 87, 94]. In contrast, our study is proactive, not only proposing a novel active learning procedure but also implementing it to actually construct a large dataset of 8,000 fully annotated CT volumes within a very short span of time, leveraging the synergy between medical professionals and AI algorithms in practice.
To overcome the deficiency of the conventional annotation method—which involves manually annotating each volume, slice, and voxel—we propose an efficient method to enable rapid organ annotation across enormous CT volumes. The efficiency has been demonstrated on 15 datasets and 8,448 abdominal CT volumes (1.2 TB in total). Our method enables high-quality, per-voxel annotations for eight organs and anatomical structures in all the CT volumes in three weeks (5 days per week; 8 hours per day), justified in §3.3. The constructed dataset is named AbdomenAtlas-8K, and its detailed statistics can be found in Figure 1. As a comparison, the conventional annotation methods [45, 26, 29, 51, 6] would take up to 1,600 weeks (30.8 years) to complete such a task<sup>3</sup>.
Our key innovation is that rather than time-consuming annotating organs voxel by voxel, we leverage existing data and incomplete labels from an assembly of 16 public datasets. AI models are trained on the labeled part of data and generate predictions for a large number of unlabeled parts of data. We then actively revise the model predictions by only selecting the most salient part of the regions for annotators to correct. Moreover, our study provides a taxonomy of common errors made by AI algorithms and annotators. This taxonomy minimizes error duplication and augments data diversity, ensuring the sustainability of continuous revision of AI algorithms and organ annotations. Our novel active learning procedure involves only one trained annotator, three AI models, and commercial software  $(\mathrm{Pair}^4)$ . After the procedure, annotators confirm the annotation of all the CT volumes by visual inspection. A large-scale, but private, dataset [92, 93] is used for external validation.
${}^{3}$  Trained annotators annotate abdominal organs at a rate of 30-60 minutes per organ per three-dimensional CT volume reported by Park et al. [65]. We have also conducted a reader study to estimate the annotation time for each CT volume in our dataset, detailed in Appendix Table 4.
<sup>4</sup>We purchased a license from Pair. Alternatively, we also used open-source MONAI Label for annotation.
In summary, we made two major contributions. Firstly, AbdomenAtlas-8K was a composite dataset that unified datasets from 26 different hospitals worldwide. In total, over  $60.6 \times 10^{9}$  voxels were annotated in comparison with  $4.3 \times 10^{9}$  voxels annotated in the public datasets. We scaled up the organ annotation by a factor of 15 and released the masks for 5,195 of the 8,448 CT volumes. This large-scale, multi-center dataset can also impact downstream clinical tasks such as surgery, treatment, abdomen atlas, and anomaly detection. Secondly, the proposed active learning procedure can generate an attention map to highlight the regions to be revised by radiologists, reducing the annotation time from 30.8 years to three weeks and accelerating the annotation process by an impressive factor of 533. This strategy can quickly scale up annotations for creating many other medical imaging datasets.
Large dataset construction. Kirillov et al. [43] created a huge natural image dataset of 1B masks and 11M images, but it lacks semantic information, and its efficacy is limited when applied to 3D volumetric medical images [52, 34]. Our AbdomenAtlas-8K, containing 8,448 CT volumes with per-voxel annotated eight abdominal organs, is the largest annotated CT dataset at the time this paper is written. We hereby review the existing public datasets that contained over 500 CT volumes with per-voxel annotated organs [23]. For example, AMOS [40], TotalSegmentator [89], and AbdomenCT-1K [55] provided 500, 1,024, and 1,112 annotated CT volumes, respectively. Both TotalSegmentator and AMOS derived their data from a single country, with the former reflecting the Central European population from Switzerland and the latter representing the East Asian population from China. In comparison, AbdomenAtlas-8K presented a greater data diversity because the CT volumes were collected and assembled from at least 26 different hospitals worldwide. While AbdomenCT-1K sourced data from 12 hospitals, AbdomenAtlas-8K contained approximately eight times the CT volumes (8,448 vs. 1,000) and twice the variety of annotated organs (8 vs. 4). Concurrently, we are actively expanding the classes covered by AbdomenAtlas-8K. Starting with the set of 104 classes found in TotalSegmentator, we aim to significantly diversify the range of classes covered.
Active learning for segmentation. Uncertainty and diversity are key criteria in active learning. Uncertainty-based criteria assess the value of annotating a data point based on the uncertainty (e.g., entropy) of AI predictions [18, 19, 56, 75, 5, 63, 11]. On the other hand, diversity-based criteria aim to select unannotated samples that differ from each other and from those already annotated [48, 25, 44, 59, 81, 82, 76]. For additional active learning methods, we refer the reader to comprehensive literature reviews [83, 62, 31, 71]; but these methods face computational complexity challenges with segmentation tasks and large unannotated data pools. To overcome this, we summarized typical errors made by humans and computers. Our active learning procedure considered the anatomical priors, uncertainty in AI prediction, and data diversity, and, importantly, pivoted a prospective application of active learning rather than retrospective studies. Moreover, the derived criteria can generate an attention map, pinpointing areas necessitating revision, thereby enabling precise detection of high-risk prediction errors (evidenced in Table 1). Consequently, our strategy markedly diminished the workload and annotation time for annotators by a factor of 533.
3 AbdomenAtlas-8K
Overview. We propose an active learning procedure comprising two components: error detection from AI predictions (§3.1) and manual revision performed by radiologists to review and edit the most significant errors detected (§3.2). By repeatedly implementing these two components, it is possible to expedite the creation of fully annotated datasets for multi-organ semantic segmentation. Finally, we describe the data construction strategies (§3.3). In this work, we have applied our approach to 8,448 CT volumes in portal (44%), arterial (37%), pre- (16%), and post-contrast (3%) phases.
3.1 Label Error Detection Revealed by Attention Maps
Figure 2 shows the process to generate attention maps for eight target organs. The attention maps can localize the potential error regions for human annotators to review and edit AI predictions.
(1) Inconsistency. To quantify inconsistency, we calculate the standard deviation of the soft predictions produced by multiple AI architectures, including Swin UNETR, nnU-Net, and U-Net. Regions
Figure 2: Attention map generation. The criteria inconsistency, uncertainty, and overlap refer to regions where model predictions diverge, exhibit high entropy values, and where multiple organ predictions overlap, respectively. The attention map visualizes a combination of these regions, drawing radiologists' attention to where AI predictions might falter. A standard color scheme helps in highlighting regions that merit closer review and revision. More examples are in Appendix Figure 11.
with high standard deviation indicate higher inconsistency and may require further revision.
$$
\text {I n c o n s i s t e n c y} _ {i, c} = \sqrt {\frac {\sum_ {n = 1} ^ {N} \left(p _ {i , c} ^ {n} - \mu_ {i , c}\right) ^ {2}}{N}}, \tag {1}
$$
where the subscript  $c$  represents class  $c$  of our eight target organs. For each voxel  $i$ ,  $p_{i,c}^{n}$  represents the soft prediction value obtained from the  $n$ -th AI architecture of class  $c$  at that voxel's index  $i$ , ranging from 0 to 1.  $\mu_{i,c}$  represents the average prediction value obtained by combining the results of three AI architectures at the same voxel index. In this study, there are three AI architectures, so  $N$  is equal to three. Then, the Inconsistency $_{i,c}$  value is determined by the standard deviation of the soft prediction values from the three AI architectures.
(2) Uncertainty. To estimate the degree of certainty linked with the AI prediction of eight target organs, we determine the entropy of the soft predictions for each organ. Regions of higher entropy values suggest diminished confidence and increased ambiguity, potentially escalating the chances of encountering prediction errors within that specific area [98], which may necessitate further revision.
$$
\text {U n c e r t a i n t y} _ {i, c} = - \frac {\sum_ {n = 1} ^ {N} p _ {i , c} ^ {n} \times \log \left(p _ {i , c} ^ {n}\right)}{N}. \tag {2}
$$
The Uncertainty  $\mathbf{\mu}_{i,c}$  is averaged over different AI architectures  $(N = 3)$ .
(3) Overlap. The overlap in organ prediction can indicate potential errors. If a voxel is predicted to be a part of both the liver and kidneys, even without ground truth, we can reasonably forecast a prediction mistake. We use the following measure to detect organ overlap in predictions.
$$
\operatorname {O v e r l a p} _ {i, c} = \left\{ \begin{array}{l l} 1 & \text {i f} p _ {i, c} ^ {n} > 0. 5 \text {a n d} \exists p _ {i, c \notin} ^ {n} > 0. 5 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
We generate pseudo labels by applying a threshold of 0.5 to the probability values. Pseudo labels refer to organ labels predicted by AI models without any additional revision or validation by human annotators. The overlap value, denoted as  $\text{Overlap}_{i,c}$ , is determined based on the following criteria: if the prediction value for class  $c$  exceeds the threshold of 0.5 for at least one AI architecture and there exists a prediction value not belonging to class  $c$  that exceeds 0.5 for the same voxel index  $i$ , then the overlap value is set to 1; otherwise, it is set to 0.
As a result, an attention map is generated to help annotators quickly locate regions that require revision or confirmation. We combine the inconsistency, uncertainty, and overlapping regions to produce the attention map. Consequently, a higher  $\text{Attention}_{i,c}$  value in the 3D attention map indicates a greater risk of a prediction error for that voxel.
$$
\text {A t t e n t i o n} _ {i, c} = \text {I n c o n s i s t e n c y} _ {i, c} + \text {U n c e r t a i n t y} _ {i, c} + \text {O v e r l a p} _ {i, c} \tag {4}
$$
To assess the attention map, we identified error regions which is the summation of all false positive (FP) and false negative (FN) areas between the ground truth annotations (available in JHH [92]) and the pseudo labels predicted by AI. We then compare the attention maps with the error regions and calculate the sensitivity and precision, reported in §4.1.
3.2 Active Learning Procedure
Algorithm. Our active learning procedure has eight steps.  $①$  Train an AI model from scratch, denoted as  $\mathcal{M}_0$ , using 2,100 CT volumes from 16 partially labeled public datasets. This takes approximately 40 hours.  $②$  Direct test the current model (e.g.,  $\mathcal{M}_0$ ) on all 8,448 CT volumes to segment eight organs. This takes around 12 hours.  $③$  Compute organ-wise attention maps for each CT volume using the criteria of inconsistency, uncertainty, and overlap ( $\S 3.1$ ), which highlight the regions that potentially have prediction errors and require human revision.  $④$  Compile a priority list sorted by the sum of attention maps. The larger the attention map is, the more urgent the CT volume requires revision.  $⑤$  Ask the annotators to review the top  $5\%$  (analyzed from Figure 4) CT volumes from the list and revise the pseudo labels guided by the attention maps.  $⑥$  Reassemble the annotation of each revised CT volume based on the label priority (detailed in the next paragraph).  $⑦$  Fine-tune the current model  $(\mathcal{M}_t)$  using the reassembled annotation to obtain  $\mathcal{M}_{t+1}$ .  $⑧$  Repeat steps  $②-⑦$  until the annotators confirm that the CT volume on the top of the prioritized list in step  $④$  does not need further revision, suggesting that AI predictions of the most important CT volumes have minimal errors.
Technical details in  $⑥$ : label priority. In the assembly process, the utmost priority is given to the original annotations supplied by each public dataset. Subsequently, we assign secondary priority to the revised labels from our annotators. The pseudo labels, generated by AI models, are accorded the lowest priority. Based on this label priority, we reassemble the labels for each CT volume. This involves three different scenarios: inherited from the original labels of the public dataset if available, revised (if our annotator confirms that the pseudo labels have errors), and unchanged (if our annotator confirms that the pseudo labels predicted by the AI are correct). Figure 3 displays examples of the reassembled annotations from AbdomenAtlas-8K.
Figure 3: AbdomenAtlas-8K annotations. The annotations for our eight target organs (opaque) have been revised by our annotator or inherited from the original annotations of the partially labeled public datasets. The remaining organs and tumors (transparent) can be used for future research.
Figure 4: Attention size distribution. The y-axis denotes the attention size, the sum of Equation 4 over eight classes; each point corresponds to a distinct CT volume. A larger attention size implies a greater need for revision in various regions. While most CT volumes exhibit a small attention size, a few notable outliers (marked in red) stand out. These outliers are of high priority for revision by human experts. According to the figure, the ratio of outliers is about  $5\%$  (highlighted in red). The  $5\%$  is estimated by the plot and also related to the budget of human revision for each Step in the active learning procedure. It is essential to emphasize that roughly  $5\%$  of CT volumes within each dataset are highly likely to contain predicted errors, requiring further revision by our annotator.
Figure 4 illustrates the attention size of CT volumes within each partially labeled public dataset. It is important to note that the BTCV [46] and AMOS22 [40] datasets already have original annotations for our eight target organs. By analyzing the curve depicting the decreasing value of the attention map, we observed that among the 5,195 CT volumes from the 16 partially labeled abdominal datasets, only
Figure 5: (a) Error taxonomy. To minimize repetitive errors during human revision, we collate and analyze common mistakes that occur during the data made by either AI or human annotators. (b) Errors in pseudo labels. For example, the cavity in the stomach is missing; the enlarged pancreas tail caused by a pancreatic tumor is missing; the enlarged gall bladder is missing. (c) Errors in human annotations. The high intra-annotator variability can be attributed to the ambiguity in defining organ or tumor boundaries, whereas the high inter-annotator variability is often a result of inconsistent annotation protocols and guidelines across different institutes.
$5\%$  of the volumes exhibited significantly large attention map values. These high values indicate a higher risk of prediction errors, suggesting the need for further revision. Extrapolating this finding to a larger dataset of 8,000 CT volumes, we can estimate that the annotator will need to confirm and revise approximately 400 CT volumes. Assuming a rate of 15 minutes per CT volume and an 8-hour workday, this process would take approximately 12.5 days to complete. The attention map values are expected to decrease after fine-tuning  $\mathcal{M}_0$  with revised labels as a new benchmark.
Figure 5(a) summarizes typical errors encountered in the active learning procedure into two categories: pseudo label errors and ground truth errors. As illustrated in Figure 5(b), pseudo label errors refer to the errors in the AI predictions. These errors often arise from irregular organ shapes, such as the absence of a cavity in the stomach or the omission of an enlarged pancreas and gall bladder. Discrepancies in CT volumes between the training and testing data, such as variations in scanners, protocols, reconstruction methods, and contrast enhancement, can also contribute to these inaccuracies. These issues result in the model's predictions lacking accuracy in representing these specific anatomical structures. As examples in Figure 5(c), ground truth errors are errors in AI model predictions that result from inaccuracies in the human annotations used for training the model. These inaccuracies may arise due to unclear organ boundaries or inconsistency in labeling protocols across different institutions, introducing variations into the model's predictions.
3.3 Dataset Construction
Annotators. Our study recruited three annotators, comprising a senior radiologist with over 15 years of experience and two junior radiologists with three years of experience. The senior radiologist undertook the task of annotation revision in the active learning procedure. Before releasing AbdomenAtlas-8K, two junior radiologists looked through the masks in the entire AbdomenAtlas-8K and made revisions if needed (i.e., our method missed the error regions). In addition, the two junior radiologists conducted the inter-annotator variability analysis (Figure 8) and recorded the time for conventional methods when each organ must be annotated voxel by voxel (Appendix Table 4).
Efficiency. Why 30.8 years? We considered an 8-hour workday, five days a week. A trained annotator typically needs 60 minutes per organ per CT volume [65]. Our AbdomenAtlas-8K has a total of eight organs and around 8,000 CT volumes. Therefore, annotating the entire dataset requires  $60 \times 8 \times 8000$  (minutes) /  $60/8/5 = 1600$  (weeks) = 30.8 (years). Why three weeks? Using our
5The  $5\%$  is empirically estimated based on (1) the observation in the distribution of the attention size (i.e., the number of outliers in Figure 4) and (2) the annotation budget at each Step in the active learning procedure. If there are many outliers or a limited budget, the threshold needs to be increased accordingly.
<sup>6</sup>Such revisions were seldom required based on our study—only 55 out of 8,448 volumes needing adjustments.
Table 1: Evaluation metrics on JHH. The sensitivity and precision are evaluated between our organ attention maps and organ error regions. Our attention map exhibits high average sensitivity and precision, indicating its effectiveness and accuracy in detecting false positives (FP) and false negatives (FN). This highlights its capability to precisely identify regions that need revision.
Metrics	Spl	RKid	LKid	Gall	Liv
Sensitivity	0.91 ± 0.25	0.93 ± 0.17	0.92 ± 0.18	0.99 ± 0.07	0.74 ± 0.33
Precision	0.68 ± 0.40	0.90 ± 0.22	0.85 ± 0.23	0.67 ± 0.33	0.88 ± 0.12
Metrics	Sto	Aor	IVC	Pan	Avg.
Sensitivity	0.98 ± 0.10	0.98 ± 0.11	0.98 ± 0.09	0.90 ± 0.21	0.93 ± 0.17
Precision	0.85 ± 0.16	0.82 ± 0.21	0.75 ± 0.22	0.91 ± 0.15	0.81 ± 0.22