SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Error region vs. attention map. The discrepancy between the model's pseudo labels and the ground truth delineates the error regions in predictions, comprising both false positives (FP) and false negatives (FN). These error regions serve as the benchmark for evaluating the sensitivity and precision of our attention map ( $\S 4.1$ ).
active learning strategy, only 400 CT volumes require manual revision from the human annotator (15 minutes per volume). That is, we managed to accelerate the annotation process by a factor of  $60 \times 8 / 15 = 32$  per CT volume. Therefore, we completed the entire annotation procedure within three weeks, as reported in the paper. Human efforts:  $400 \times 15$  (minutes) /  $60 / 8 = 12.5$  (days) plus the time commitment for training and testing AI models taking approximately 8.5 (days).
Dataset splits. For 8,448 CT volumes in AbdomenAtlas-8K, we split them into training (5500 CT volumes), validation (500), and test (2448) sets. Each volume contains per-voxel annotations of the spleen, liver, left& right kidneys, stomach, gallbladder, pancreas, aorta, and IVC. Note that the JHH dataset is a proprietary, multi-resolution, multi-phase collected from Johns Hopkins Hospital [92, 42, 100, 47]. This dataset is used for external validation, including the assessment of the attention map quality (Table 1), AI generalizability (Table 2), the improvement of label quality in the active learning procedure (Appendix Table 5), and the performance of AI trained on a combination of public datasets and our AbdomenAtlas-8K(Appendix Table 6).
4 Experiment & Result
4.1 Attention Map and Annotation Evaluation
Evaluation of attention map. Our attention map is evaluated on 1,000 CT volumes of the JHH dataset [92], which is not included in the training process. The evaluation is performed across the eight target organs. Once an error is detected, it counts as a hit; otherwise, as a miss. To evaluate the quality of the attention map, two metrics are used: Sensitivity  $= \mathrm{TP} / (\mathrm{TP} + \mathrm{FN})$  and precision  $= \mathrm{TP} / (\mathrm{TP} + \mathrm{FP})$ , where a true positive (TP) means the attention map found real mistakes the AI made; a false negative (FN) means it missed some of the AI's mistakes; and a false positive (FP) means it found mistakes where the AI was actually right. Sensitivity and precision are all calculated at the volume (a group of voxels) level rather than the voxel level. We chose sensitivity and precision because this experiment is designed to evaluate an error detection task rather than a segmentation task (comparing the boundary of attention maps and error regions). They can measure how well the attention maps detect the real error regions and whether the errors being detected are real errors, respectively. The results of the attention map evaluation are presented in Table 1. The mean values of sensitivity and precision metrics for our attention maps, with respect to eight target organs, are  $0.93 \pm 0.17$  and  $0.81 \pm 0.22$ . Figure 6 shows the error regions and our attention map. These results
Figure 7: (a) Pseudo vs. revised labels. Pseudo labels are predicted by AI and then revised by annotators based on attention maps, resulting in revised labels. (b) AI predictions before vs. after fine-tuning. The post-fine-tuning results exhibit superior accuracy in segmenting the organ compared to the pre-fine-tuning result. Appendix Table 5 quantifies the DSC scores before and after fine-tuning.
(a) Al architecture variability
(b) Inter-annotator variability
Figure 8: The DSC score in each cell is calculated between the corresponding row and column. (a) AI architecture variability. The segmentation predictions made by different AI architectures display minor variations for eight organs, as evidenced by the corresponding DSC scores. Overall, the AI predictions align closely with the annotations provided by human experts. (b) Inter-annotator variability. The DSC scores between AI predictions and human annotators consistently outperform the score between the two human annotators, suggesting a comparable annotation quality between AI predictions and human annotations in the segmentation of six organs.
suggest that our attention map effectively captures the false positives and false negatives in error regions, demonstrating a high degree of accuracy in identifying prediction errors.
Evaluation of human annotation quality. We provided the visualization of pseudo labels (generated by AI) and revised labels (annotated by humans) in Figure 7(a). The revised labels were used to fine-tune the AI; we then compared the pseudo labels predicted by the AI before and after fine-tuning for the same CT volume from the JHH dataset, as shown in Figure 7(b). After fine-tuning the AI using the revised labels, the AI was able to accurately segment the entire stomach on the previously unseen CT volumes. This demonstrated the high quality of the revised labels and their efficacy in enhancing AI performance. We further quantified the human annotation quality measured by Dice Similarity Coefficient (DSC) and normalized surface Dice (NSD) using 1,000 CT volumes from the JHH dataset. The results in Appendix Table 5 show continual improvements in label quality along the active learning procedure. For example, AI models exhibited a marked improvement in the (pseudo) labels of aorta and postcava, jumping from  $72.3\%$  to  $83.7\%$  and  $76.1\%$  to  $78.6\%$ , respectively.
4.2 Label Bias and Segmentation Quality Evaluation
Evaluation of label bias. We first evaluate the predictions made by three AI architectures, i.e., Swin UNETR, nnU-Net, and U-Net, on CT volumes derived from the JHH dataset [92]. The segmentation predictions of these architectures are illustrated in Appendix Figure 10. We use the original annotations of the JHH dataset [92], which include our eight target organs. Consequently, for each CT volume, we have four predictions: three from the AI models and one from human experts. We then compute the DSC score between each pair across eight organs. These results are presented in Figure 8(a) and demonstrate that while the three AI architectures produce slightly varied predictions, they closely resemble those performed by human experts. To prevent label bias to a specific architecture, the final annotations in our AbdomenAtlas-8K are determined by averaging the three AI predictions. Our approach contrasts with the TotalSegmentator dataset [89], which exclusively relies on nnU-Net for segmentation. This could potentially generate a biased dataset that impedes model generalization across different architectures or scenarios.
Evaluation of AI prediction quality. To assess the automated annotation quality in AbdomenAtlas-8K, we asked for the assistance of two additional human annotators to help us modify the pseudo labels of our target eight organs. Due to time limitations, they are only able to revise six out of the eight organs, specifically on the 17 CT volumes from the BTCV dataset [46]. We compute DSC scores comparing AI predictions with those annotated independently by two human annotators, referred to as Dr1 and Dr2. Figure 8(b) presents the results for six organs. The DSC scores between AI predictions and each human annotator (Swin UNETR vs. Dr1 or Swin UNETR vs. Dr2) consistently exceed the score between the two human annotators (Dr1 vs. Dr2). These findings suggest that the automated AI annotations fall within the range of inter-annotator variability, thereby indicating that the quality of our automated annotations is comparable to human annotations.
4.3 Benchmarking
AbdomenAtlas-8K enables precision medicine for various downstream applications. We showcased one of the most pressing applications—early detection and localization of pancreatic cancer, an extremely deadly disease with a 5-year relative survival rate of only  $12\%$  in the United States. The AI trained on a large, private dataset at Johns Hopkins Hospital (JHH) performed arguably higher than typical radiologists [92, 47, 42]. But this AI model and annotated dataset were inaccessible due to the many policies. Now, our paper demonstrated that using AbdomenAtlas-8K (100% made up of publicly accessible CT volumes), AI can achieve similar performance when directly tested on the JHH dataset (see Table 2). This study is a concrete demonstration of how AbdomenAtlas-8K can be used to train AI models that can be generalized to many CT volumes from novel hospitals and be adapted to address a range of clinical problems. For a more in-depth analysis of the segmentation performance of AI models, we present the comprehensive category-wise scores comparison across eight organs in Appendix Table 7.
Table 2: Benchmark results of AI models trained on AbdomenAtlas-8K. We directly apply four AI models, i.e. SwinUNETR, UNETR, U-Net, and SegResNet, trained on AbdomenAtlas-8K (public) to JHH (unseen, private) and compared its performance with AI trained on JHH. The AI models trained on AbdomenAtlas-8K exhibit comparable results in average mDSC and mNSD across eight organs when compared with AI models trained on the JHH dataset, proving the high generalization capacity of AI models trained on AbdomenAtlas-8K.
AI Models	Trained on JHH (private)	Trained on AbdomenAtlas-8K (public)
mDSC (%)	mNSD (%)	mDSC (%)	mNSD (%)
SwinUNETR [84]	84.8 ± 12.6	66.5 ± 14.8	86.5 ± 7.5	60.8 ± 10.9
UNETR [28]	78.6 ± 13.0	53.9 ± 14.1	86.6 ± 6.5	59.5 ± 10.4
U-Net [73]	84.8 ± 11.2	64.2 ± 14.8	87.3 ± 6.0	61.0 ± 10.2
SegResNet [12]	87.6 ± 6.60	64.9 ± 10.9	87.0 ± 6.1	60.4 ± 10.1
5 Discussion
Impact. The scientific community has generally agreed that large volumes of annotated data are required for developing effective AI algorithms [39, 2, 60, 24, 27, 101]. For example, developing
Foundation Models for healthcare has recently raised much attention. The Foundation Model refers to an AI model that is trained on a large dataset and can be adapted to many specific downstream applications. This requires a large-scale, fully-annotated dataset. The currently available medical datasets are too small to represent the real data distribution in clinics [80, 13, 68]. The availability of large-scale, multi-center, fully-labeled data (summarized in Appendix Table 3) is one of the most significant cornerstones for both the development and evaluation of CADe systems [90, 103]. In recent years, the rise of imaging data archives [74, 57, 38, 3, 14, 16, 58, 97, 49, 99] and international competitions [45, 86, 78, 79, 30, 29, 1, 15, 4, 40] produced several publicly available datasets for benchmarking AI algorithms, but these datasets usually are of small size, contain partial labels, come from various scanners & protocols, and are therefore often limited in their scope [22, 50, 102, 95, 96, 77, 67, 88, 42]. We anticipate that our AbdomenAtlas-8K can play an important role in enabling the model to capture complex organ patterns, variations, and features across different imaging phases, modalities, and a wide range of populations. This has been partially evidenced in Appendix Table 6, wherein we generalize AI to CT volumes taken from different hospitals.
Limitation. Pseudo-labels have the potential to expedite tumor annotation procedures as well, but the risk of producing a large number of false positives is a significant concern. The presence of false positives could significantly increase the time annotators need to accept or reject a detection. In total, our AI models generate 51,852 tumor masks, including tumors in the colon, liver, hepatic vessel, pancreas, kidneys, and lung. However, only using AbdomenAtlas-8K, we are not able to assess the AI performance of tumor detection due to the lack of comprehensive pathology reports or expert annotations to describe the tumors in the majority of these publicly available CT volumes. To address this problem, we collected 392 tumor-free (private) CT volumes from Johns Hopkins Hospital to evaluate the false positives in the pseudo labels. Using kidney tumor detection as an example, there are 37 out of 392 CT volumes containing false positives  $(\mathrm{FPR} = 9.4\%)$  and a total of 161 false positives in the 37 CT volumes. Furthermore, we stratified the dataset based on the type of blood vessels, identifying average false positive rates of  $11.83\%$  and  $4.6\%$  for the venous and arterial vasculature, respectively. Therefore, while pseudo labels can be helpful, we anticipate a more effective method for false positive reduction is needed to enable a clinically practical tumor annotation, given the complexity of tumors compared with organs. As an extension, we plan to add tumor annotations to AbdomenAtlas-8K in three possible directions. Firstly, we plan to recruit more experienced radiologists to revise tumor annotations. Secondly, we will incorporate the pathology reports (based on biopsy results) into the human revision. These actions can reduce potential label biases and label errors from human annotators. Thirdly, we will exploit synthetic data (tumors) that can produce enormous tumor examples and their precise masks for AI training and validation [32, 47, 33].
In TotalSegmentator, the labels were largely generated by a single nnU-Net re-trained continually. Depending solely on nnU-Net could introduce a potential label bias favoring the nnU-Net architecture. This means that whenever TotalSegmentator is employed for benchmarking, nnU-Net would always outperform other segmentation architectures (e.g., UNETR, TransUNet, SwinUNETR, etc.). This observation has been made in several publications such as Huang et al. [35]. In contrast, our AbdomenAtlas-8K incorporates predictions from three different AI architectures, preventing bias towards one specific architecture [89]. However, such a solution comes with increased computational costs, and the performance of the AI architectures can vary. Taking an average of the predictions may result in the final outcome being pulled down by poorly performing AI architectures.
Our study shows the effectiveness of an active learning procedure, which combines the expertise of an annotator with the capabilities of a trained AI model. It not only deploys multiple AI models to detect prediction errors but also prompts annotators to revise these potential failures. Using this approach, we successfully annotated 8,448 abdominal CT volumes of different organs within three weeks, expediting the annotation process by a staggering factor of 533. Furthermore, our experiments demonstrate that the interaction between human and AI models can produce comparable or even superior results than that of a human annotator alone. This indicates that our approach can leverage the strengths of both human and AI models to achieve accurate and efficient annotation. Leveraging our efficient annotation framework, we anticipate that larger-scale medical datasets of various modalities, organs, and abnormalities can be curated at a significantly reduced cost, ultimately contributing to the development of Foundation Models in the medical domain [91, 52, 61].
This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research and the Patrick J. McGovern Foundation Award. We appreciate the effort of the MONAI Team to provide open-source code for the community. This work has partially utilized the GPUs provided by ASU Research Computing. We thank Elliot K. Fishman, Linda Chu, and Satomi Kawamoto for providing the JHH dataset for external validation; Yuxiang Lai for generating the 3D rendering of segmentation; thank Xiaoxi Chen for reviewing and revising AI predictions; thank Seth Zonies and Andrew Wichmann for providing legal advice on the release of AbdomenAtlas-8K; thank Yu-Cheng Chou, Jieneng Chen, Junfei Xiao, Wenxuan Li, and Xiaoding Yuan for their constructive suggestions at several stages of the project. The content and dataset of this paper are covered by patents pending.
[102] Jianpeng Zhang, Yutong Xie, Yong Xia, and Chunhua Shen. Dodnet: Learning to segment multi-organ and tumors from multiple partially labeled datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1195–1204, 2021.
[103] Zongwei Zhou. Towards Annotation-Efficient Deep Learning for Computer-Aided Diagnosis. PhD thesis, Arizona State University, 2021.
[104] Zongwei Zhou, Michael B Gotway, and Jianming Liang. Interpreting medical images. In Intelligent Systems in Medicine and Health, pages 343-371. Springer, 2022.
[105] Zongwei Zhou, Jae Shin, Ruibin Feng, R Todd Hurst, Christopher B Kendall, and Jianming Liang. Integrating active learning and transfer learning for carotid intima-media thickness video interpretation. Journal of digital imaging, 32(2):290-299, 2019.
[106] Zongwei Zhou, Jae Shin, Lei Zhang, Suryakanth Gurudu, Michael Gotway, and Jianming Liang. Finetuning convolutional neural networks for biomedical image analysis: actively and incrementally. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7340-7351, 2017.
[107] Zongwei Zhou, Jae Y Shin, Suryakanth R Gurudu, Michael B Gotway, and Jianming Liang. Active, continual fine tuning of convolutional neural networks for reducing annotation efforts. Medical image analysis, 71:101997, 2021.