Subgroup performance of a commercial digital breast tomosynthesis model for breast cancer detection

Domain: biology
Published: 2026-03-19
Editorial outcome: minor_revision

Abstract
AI models show potential to improve breast cancer screening, however detailed subgroup evaluations to uncover the strengths and weaknesses of models are lacking. This study presents a granular evaluation of a commercial AI model for cancer detection on digital breast tomosynthesis (DBT) on a retrospective cohort of 167,860 screening exams in female patients. Performance in distinguishing screen detected cancers (1,368 exams) from negative exams (166,387 exams) is stratified across demographic, imaging, and pathologic subgroups to identify disparities. The overall AUROC is 0.91 and sensitivity is 0.73 with robust performance across demographics. In-situ cancers (AUROC: 0.85, sensitivity: 0.55), calcifications (AUROC: 0.80, sensitivity: 0.66), and dense breast tissue (AUROC: 0.88, sensitivity: 0.63) are associated with lower performance, while masses (AUROC: 0.93, sensitivity: 0.85) and architectural distortions (AUROC: 0.90, sensitivity: 0.83) are associated with higher performance. These results highlight the need for detailed evaluations and vigilance in adopting new clinical tools.

Introduction
Breast cancer is the most common type of cancer in women and causes 42,000 deaths each year in the United States [1]. Screening mammography has historically been associated with a 38–48% reduction in breast cancer mortality through early detection of abnormalities and enabling earlier treatment [2–4], although recent analyses of contemporary observational data suggest that the observed protective effects may not be solely attributable to screening itself [5]. The Breast Imaging Reporting and Data System (BI-RADS), developed by the American College of Radiology, is a standardized framework used to categorize breast imaging findings and guide management. BI-RADS includes both an overall assessment score (ranging from 0 to 6) and specific lesion descriptors such as mass shape, margin, calcification morphology, and asymmetry type. Each BI-RADS category corresponds to a recommended clinical action: for example, Category 1 (negative) and 2 (benign) require routine screening, Category 3 (probably benign) suggests short-interval follow-up, and Categories 4 and 5 (suspicious and highly suggestive of malignancy) prompt biopsy [6]. Despite this, the sensitivity and specificity of 2D screening mammography remain around 86.9% and 88.9% resulting in unnecessary recalls and biopsies, as well as missing cancers [7]. Earlier data from the Breast Cancer Detection Demonstration Project (BCDDP) also showed age-related sensitivities of 90% for women under 50 and 95% for women 50 and older [8].

Over the past two decades, digital breast tomosynthesis (DBT) has emerged as an advanced imaging modality designed to address the limitations of 2D mammography. DBT provides pseudo-three-dimensional imaging of the breast by reconstructing multiple 2D projections into image slices and can improve specificity by reducing the masking effect of overlapping tissues [9,10]. These advantages have led to widespread adoption of the technology since its FDA approval in 2011 [11], however research has demonstrated weaknesses in some areas compared to conventional mammography. These include the increased interpretation times of DBT images – often up to twice as long as 2D mammography–and their potential for reduced visibility of calcifications, a key indicator for certain types of breast cancer [12–14].

Over the past several years, multiple deep learning models have been developed both in the research and commercial settings for diagnosis of breast cancer on 2D mammography. A recent meta-analysis demonstrated AI model performance for 2D mammography outperformed radiologists (AUC: 0.87 vs 0.81, P = 0.002) in reader studies [15]. However, reader studies are difficult to translate to real-world experience due to a manufactured reading environment and upsampling of abnormal cases. In purely retrospective studies, AI model performance for 2D mammography was not shown to be better than radiologists (AUC: 0.89 vs 0.96, P = 0.152) [15]. For DBT, three reader studies and one retrospective study demonstrated generally higher area under the receiver operating characteristics curve (AUC) for AI compared to radiologists, however with wide confidence intervals [15–19].

Little work currently exists that studies the performance of breast AI models across various subgroups of patient demographics, imaging characteristics, and pathology subtypes. For AI to gain a foothold in clinical care, clinicians must be able to understand conditions under which models can fail [20–24]. Broad interpretation of model performance across all cases can be misleading, since a model that detects the same cancers as a radiologist is not clinically meaningful. Rather, the radiologist and AI model should augment one another with complementary strengths to yield a net positive effect in detection and reduction of false positives [25–27].

In this study, we evaluate a commercial AI model for breast cancer detection on digital breast tomosynthesis (DBT) on a large retrospective cohort of 167,860 screening mammography exams from 61,332 women [28], and compare its performance to previously published metrics as well as its FDA validation study. Similar to prior work, we examine model performance across demographics, including race, ethnicity, and age groups. Additionally, performance is considered across clinically meaningful subtypes of cancer (invasive, DCIS) and subtypes of negative exams (i.e. negative at screening, negative on diagnostic, negative on biopsy). Further, we evaluate performance across finding types (mass, calcification, architectural distortion, and asymmetry) and across specific BI-RADS imaging descriptors.

Results
A total of 166,387 negative exams and 1,368 positive exams were included in the final evaluation of the model (Figure 1). A summary of the distribution is shown in Table 1. The majority of exams were for Black (77,306, 46.1%) and White patients (72,498, 43.2%), with 653 (47.7%) and 619 (45.2%) screen-detected cancers, respectively. Non-Hispanic or Latino patients (139,088, 82.9%) represented the major ethnicity. Breast density was distributed as A (18,282, 11.0%), B (68,861, 41.3%), C (70,438, 42.2%), and D (9229, 5.5%).

2.1 Overall Model Performance
The model achieved an overall AUC of 0.91 (95% CI: 0.90–0.92) in distinguishing between screen-detected cancers and negative exams, with a recall of 0.73, FPR of 0.07, TNR of 0.93, and FNR of 0.27 (Table 2). Performance was not statistically different across Race, Ethnicity, or Age. In addition, model performance was evaluated across multiple subgroups including screening outcome, breast density, imaging features, and pathology subtype.

2.2 Model Trends by Outcome Subgroups
We observed that model prediction scores trended slightly higher between screen negative, diagnostic negative, and confirmed benign exams, with decreasing percentages of exams being classified as normal according to the operating point of 0.1 (Figure 2). The model classified 93% (137,240/146,903) of screening studies as negative, 90% (14,058/15,557) of diagnostic negative exams as negative, and 81% (3,191/3,927) biopsy proven benign exams as negative. The model classified 73% (1,002/ 1,368) of screen-detected cancers as positive. Finally, the model classified 31% (33/105) of interval cancers as positive.

2.3 Model Trends by Pathological Subgroups
For binary classification, we considered invasive cancers and Ductal Carcinoma In Situ (DCIS) as positive, and all other cases as negative (screen negative, diagnostic negative, benign, borderline, and high-risk lesions). We observed that model prediction scores trended higher as pathology subtypes became more abnormal or at higher risk of becoming cancer. At the operating point of 0.1, the model correctly classified 84% (2,634/3,132) of screening studies with a benign lesion, 74% (26/35) of studies with a borderline lesion and 70% (531/760) of studies with a high-risk lesion as negative. For screening studies with DCIS and invasive cancer outcomes, the model correctly identified 55% (248/454) of DCIS and 82% (754/914) of invasive cancers (Figure 3). Performance was generally good across invasive cancer subtypes but trended lower for certain rare subtypes. Invasive Ductal Carcinoma, Not Otherwise Specified (IDC NOS) represented the majority of cases (544/734), and had 86% (468/544) of cases classified as positive. Invasive Lobular Carcinoma (ILC) was classified correctly in 82% (82/100) of cases and Invasive Mammary Carcinoma (IMC) in 84% (56/67) of cases (Figure 4). Note that the aforementioned counts consider the number of invasive pathologies present, which differs from the total number of screen detected invasive cancers exams as some contain multiple invasive pathologies.

Model performance was significantly worse for DCIS (AUC 0.85, 95% CI: 0.83-0.87; Recall 0.55, 95% CI: 0.50-0.59; n=454; p<0.001) compared to invasive cancers (AUC 0.94, 95% CI: 0.93–0.95; Recall 0.83, 95% CI: 0.80-0.85; n=914; p<0.001) across all DCIS sub-types and grades (Figure 5). Ductal Carcinoma In Situ, Not Otherwise Specified (DCIS NOS) represented the majority of cases (201/361) and had 55% (111/201) of cases classified as positive. Note that the aforementioned counts consider the number of DCIS pathologies present which will differ than the total number of screen detected DCIS, as some exams contained multiple DCIS pathologies.

2.4 Model Trends by Imaging Features
Model performance varied based on the presence of specific findings in screening studies (Figure 6). Because exams could contain multiple findings and model results are returned at the exam level, our results are aggregated for the presence of any imaging finding at the study level. For example, if an exam has both a mass and calcification, the model output is attributed to both the mass and calcification subgroups. 84% (17,533) of abnormal screening exams had only one BI-RADS 0 lesion, while 16% (3319) of abnormal exams contained more than one. For exams containing a mass, the model correctly classified 85% (2368 of 2780) of cases in the screen negative group, 90% (2484 of 2766) in the diagnostic negative group, 84% (582 of 695) in the confirmed benign group, and 85% (196 of 231) in the screen-detected cancer group. For studies with asymmetry, the model correctly classified 92% (2799 of 3057) in the screen negative group, 91% (7992 of 8737) in the diagnostic negative group, 83% (1206 of 1449) in the confirmed benign group, and 78% (414 of 532) in the screen-detected cancer group. For studies with architectural distortion, the model correctly classified 72% (83 of 116) of cases in the screen negative group, 87% (1245 of 1431) in the diagnostic negative group, 58% (204 of 352) in the confirmed benign group, and 83% (177 of 213) in the screen-detected cancer group. For studies with calcification, the model correctly classified 80% (1735 of 2173) of cases in the screen negative group, 85% (1873 of 2196) in the diagnostic negative group, 79% (1381 of 1741) in the confirmed benign group, and 65% (408 of 623) in the screen-detected cancer group. Outputs for imaging features were further stratified by BI-RADS descriptors such as mass shapes and margins (Supplementary Figures S1 and S2), calcification distribution and morphologies (Supplementary Figures S3, S4, and S5), and architectural distortion and asymmetry subtypes (Supplementary Figures S6 and S7).

Qualitatively, exams with masses showed the best separation between model prediction scores of positive and negative exams whereas exams with architectural distortion showed the least separation. Examples of true positive, false positive, and false negative model predictions by lesion type are shown in Figure 7.

Methods
4.1 Ethics Statement
This study was approved by the Institutional Review Board of Emory University (IRB STUDY00006509). Informed consent was waived for the use of anonymized, retrospective data.

4.2 Model
The INSIGHT DBT (engine version v.1.3.0; product version v.1.1.2.0) model from Lunit, Inc (Seoul, South Korea) was used for inference in the study. The model received FDA 510k clearance in 2023 for use on screening exams from Hologic and GE scanners and is deployed across 16 countries and 44 sites. The model architecture has 23.4 million trainable parameters and consists of a ResNet34 feature extractor and an aggregation module that outputs an image-level malignancy score and a heatmap for localization. The analysis was conducted using exam-level scores which were derived by selecting the highest predicted image malignancy score for each exam. It was trained on approximately 160,000 exams from 2 countries, and 29% were cancers. 87% of the exams were from the USA and 13% from South Korea. 21% of the exams were labeled normal, 50% were benign and 29% were cancer. Of the exams labeled cancer, 78% were invasive. 47% of the exams had a BI-RADS breast density of A or B, rest 53% had BI-RADS C or D. 4% of the patients were Black, 9% Hispanic, 85% White. 75% of the exams had patient age between 40 and 79, both inclusive. The model has a set operating threshold of 0.1 which was selected by Lunit to balance sensitivity and specificity across various populations and cannot be adjusted after regulatory clearance.

4.3 Data
This analysis was conducted using digital breast tomosynthesis (DBT) images from the Emory Breast Imaging Dataset (EMBED) [28]. The relevant images were collected from screening exams in four hospitals in the Emory Healthcare network between January 2013 and December 2020. Women with both average and above-average breast cancer risk were included in the study as the INSIGHT DBT model does not specify an eligible population. Within EMBED, approximately 72.1 percent of screening exams are ComboHD studies that contain both 2D and DBT+synthetic 2D images, whereas the remainder are 2D only. For this analysis, exams that were 2D-only were excluded. No exams contain DBT or C-view only. 99.8 percent of exams are from Hologic scanners.

In total, 229,207 exams for 80,331 patients were evaluated by the model, with 1 exam failing to process. The failure was due to issues in reading or transforming the image data, stemming from missing or malformed DICOM information. Predicted abnormality scores were aggregated to the exam-level by taking the maximum score for each. 167,860 exams for 61,332 patients had unambiguous ground truth data and follow-up available and were considered in the final analysis after excluding 61,368 exams during the label assignment and data handling steps shown in Figure 1.

4.4 Ground Truthing
The original radiology report, inclusive of any addendums, was used as ground truth for imaging features and BI-RADS characteristics. Pathology ground truth was obtained from reports and summarized into six categories based on prior consensus from an expert panel of breast pathologists, oncologists, and radiologists [28]: invasive cancers, DCIS, high-risk lesions, borderline lesions, benign lesions, and non-breast cancers. The most common pathologies assigned to each category are listed in Supplementary Table S2. Demographics were obtained from the EHR.

4.5 Label Assignment
Exams were categorized using a comprehensive set of clinical conditions to determine ground truth labels for evaluation. As all images used in the evaluation were captured at screening, any follow-up diagnostic assessments (recall studies) were matched to their preceding exams. Diagnostic exams for problem evaluations were not considered, except in the case of interval cancers as described below. Descriptions of the five exam labels considered in the analysis are shown in Table 3.

The primary clinical features for label assignment were BI-RADS assessments and pathology results. Because EMBED contains data at the finding level (i.e. a screening exam may have multiple findings in each breast), finding-level assessments were aggregated to the exam-level by selecting the most severe BI-RADS assessment and pathology results from each exam to match the aggregated outputs from the INSIGHT DBT model.

4.5.1 Negative Exams
Three types of ‘negative’ exams were defined: screen negatives (SN), diagnostic negatives (DN), and biopsy-proven benign (B). Screen negatives were defined as screening exams with only BI-RADS 1 or 2 findings that did not fit the interval cancer definition. Diagnostic negatives were defined as screening exams with at least one BI-RADS 0 finding that prompted a diagnostic follow-up which only had BI-RADS 1, 2, or 3 findings. Biopsy-proven benign were defined as abnormal screening exams that prompted a diagnostic follow-up with a BI-RADS 4 or 5 finding which was biopsied and resulted in a high-risk, borderline, or benign lesion. High-risk lesions that were resected and did not result in a cancer were considered negative; those that were resected and yielded a cancer were considered in the positive group (see below). Non-breast cancers like lymphoma and other primary source of cancers were excluded in this evaluation.

4.5.2 Positive Exams
Positive exams were classified as screen-detected cancer detected via the normal screening pathway: an abnormal screening exam that prompted a diagnostic follow-up with a BI-RADS 4 or 5 finding followed by an invasive cancer or DCIS on biopsy or resection. This included high-risk lesions on biopsy that were resected and found to be malignant.

4.5.3 Interval Cancers
Interval cancers are defined as a negative screening exam followed by a confirmed cancer diagnosis within 12 months of the screening exam. These were considered as a separate class during evaluation due to the difficulty in categorizing them as either negative or positive. Some interval cancers may be visible on preceding study in retrospect while others were not. However, previous work has shown that AI models were able to detect at least some of these cancers [34–36].

4.5.4 Excluded/Invalid Exams
Because EMBED relies on human entered data, there are rare circumstances of idiosyncratic or missing data. This includes missing or invalid BI-RADS scores based on exam type, abnormal screening studies with no diagnostic follow-up, or BI-RADS 4 and 5 diagnostic studies with no biopsy results. There are also instances in which a patient leaves the healthcare system and therefore cannot be accurately ground-truthed.

Considering this, exams were excluded from the analysis for several reasons. These included: exams with incorrect data, such as screening exams with diagnostic-only BI-RADS assessments or invalid exam accessions; exams with missing data, such as abnormal screening exams with no diagnostic follow-up, screen negatives with no long-term follow-up to confirm their ground truth, or diagnostic BI-RADS 4/5 cases with no biopsy results; exams falling outside the study cohort of interest, such as diagnostic exams and exams containing previously-known breast cancers (assigned a BI-RADS 6) or non-breast cancers. Numbers and types of excluded cases are shown in Figure 1.

4.6 Analysis
4.6.1 Feature Selection for Subgroup Analysis
Model performance was assessed across various demographic, imaging, and pathologic subgroups. Demographic subgroups were considered for patient race, ethnicity, and age. The analysis included the patient race categories: Asian, Black, White, unknown or unavailable, and other patient races. The ethnicity groups considered were: Hispanic or Latino, not Hispanic or Latino, and unknown. Patient age at the time of exam was binned into three groups: under-50, 50-to-75, and over-75.

Imaging features like exam-level presence of masses, asymmetries, architectural distortions, and calcifications were also considered during analysis. Outputs for imaging features stratified by BI-RADS descriptors [6] are included in the supplement for mass shapes and margins (Supplementary Figures S1 and S2), calcification distribution and morphologies (Supplementary Figures S3, S4, and S5), and architectural distortion and asymmetry subtypes (Supplementary Figures S6 and S7).

Lastly, pathological subgroups were defined based on the final diagnosis determined from biopsy or, when available, subsequent surgical resection. For example, if a patient had an abnormal screening exam followed by diagnostic mammogram and biopsy that resulted in ductal carcinoma in situ (DCIS), followed by resection resulting in invasive ductal carcinoma (IDC), the diagnosis attributed to the screening exam would be IDC. Any subsequent post-treatment diagnoses or recurrences were not considered. Results were stratified by major pathology types: invasive cancers, DCIS, high-risk lesions, borderline lesions, and benign lesions/normal tissue. Invasive cancers and DCIS were further decomposed into common subtypes.

4.6.2 Statistics & Reproducibility
Using the ground truth categories determined during label assignment, the model was evaluated as a binary classifier with screen-detected cancers as the positive class (assigned a numeric label of 1) and the negative groups as the negative class (assigned a numeric label of 0). Interval cancers were not considered during metric calculation. Threshold-dependent metrics were evaluated at the commercial operating point for InsightDBT of 0.10. Due to the heavy class imbalance of the data towards the negative class, performance was primarily evaluated with precision, recall, false positive rate (FPR), true negative rate (TNR), false negative rate (FNR), and area under the receiver operating characteristic curve (AUROC).

Confidence intervals for all classification metrics were estimated using non-parametric bootstrapping (5,000 resamples). Statistical comparisons between subgroups were conducted using the Kruskal-Wallis test to bootstrapped AUC distributions to assess for global differences. When significant heterogeneity was observed (p < 0.05), post-hoc pairwise comparisons between subgroups were performed using bootstrapped difference tests. Where multiple group comparisons were performed, the Bonferroni correction was applied to control the family-wise type 1 error rate at 5%. When comparing AUC between invasive cancers and DCIS, both groups were compared to the same set of negative cases. Results were described as statistically significant if they met the pre-defined cutoff after correcting for multiple comparisons. All statistical analyses were conducted using Python. No statistical method was used to predetermine sample size since this was a retrospective study.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Summary\nIn this retrospective validation study, the authors evaluate the performance of a commercial AI based CAD tool (Lunit\nINSIGHT DBT) for breast cancer detection on DBT. The analysis includes more than 163,000 screening mammograms from\nthe Emory Breast Imaging Dataset and examines model performance across demographic, pathologic, and imaging\nsubgroups. The model attained an overall AUC of 0.91. However, significantly lower performance was observed in certain\nsubgroups, including cases with DCIS, calcifications, and dense breast tissue. The authors conclude that the model\ndemonstrates strong overall performance but that careful consideration of subgroup-specific limitations is essential for its\nclinical implementation.\nStrengths\nA. There is limited research on the performance of AI-based CAD models for breast cancer detection across demographic,\npathologic, and imaging subtypes. Most studies focus solely on overall performance metrics without addressing subgroup-\nspecific variations.\nB. A strength of the study is its detailed failure analysis, which offers valuable insights into the model's limitations and error\npatterns.\nC. The analysis includes more than 160,000 screening exams from a diverse population.\nWeaknesses\n1. One of the key findings is that Lunit INSIGHT DBT showed significantly lower performance in detecting non-invasive\ncancers and calcifications. This is expected, as non-invasive cancer (DCIS) typically presents as calcifications. The reduced\nperformance is likely due to the use of the DBT-specific algorithm rather than the 2D algorithm; currently, the two Lunit\nalgorithms for DBT and 2D mammography are not integrated. Since the 2D algorithm is likely better suited for detecting\ncalcifications, it is more effective in identifying cases of DCIS.\n2. The model’s sensitivity for screen-detected cancers appears relatively low at 73%, which may be partially attributed to its\nlimited sensitivity for detecting DCIS. Notably, this overall sensitivity is lower than that reported in a previous publication on\nLunit INSIGHT DBT (Radiol Artif Intell 2024;6(3):e230318). The authors are encouraged to address this discrepancy.\n3. Throughout the manuscript, the authors are encouraged to use the term DCIS in place of non-invasive cancer, as DCIS is\nthe more commonly recognized and clinically specific terminology in the breast imaging and oncology literature.\n4. Lesion-specific analyses were not performed. While model-level classifications are reported, it remains unclear whether\nthe model accurately localized the corresponding lesions.\n5. A large number of exams were excluded from the analysis. Specifically, among the 31,379 abnormal screening exams,\nthe authors are asked to clarify why a significant portion (10,310 exams) did not have diagnostic follow-up.\n6. The finding types (mass, asymmetry, architectural distortion, or calcifications) appear to be determined based on the initial\nscreening mammogram report. The authors should comment on the potential impact of interobserver variability in radiologist\ninterpretations on the reliability of these findings. Furthermore, the final finding type in the diagnostic report may differ from\nthe finding type in the screening report (e.g., architectural distortion on diagnostic examination but focal asymmetry on\nscreening examination).\nSpecific Comments\n7. Abstract: Please clarify what is meant by “various negative exam types.”\n8. Abstract: Please clarify if “non-invasive cancers” refer to ductal carcinoma in situ (DCIS). Lobular carcinoma in situ (LCIS) is considered a high-risk lesion.\n9. Background, lines 12-13: It is stated that the “sensitivity and specificity of screening 2D mammography remains between\n85-90%.” I would suggest reporting sensitivity and specificity separately.\n10. Background, line 19: Please note that DBT is not truly 3D imaging.\n11. Data: It is stated that 72.1% of screening exams are “2D + DBT + synthetic 2D” – this should be stated as DBT + 2D or\nsynthetic 2D.\n12. Data: It is stated that 72.1% of screening exams are 2D + DBT, and the remainder are 2D only. Please clarify that 2D\nonly exams are excluded from the analysis.\n13. Ground Truthing: Please clarify if “non-invasive cancers” refer to DCIS. Also, please clarify the difference between high-\nrisk lesions and borderline lesions.\n14. Analysis, lines 195-196: Please clarify/correct: “Lastly, pathological subgroups examined the final diagnosis resulting\nfrom biopsy or subsequent resection.”\n15. Outcome subgroups: The sensitivity of the model for screen-detected cancers seems quite low (73%). The sensitivity of\nthe model for the detection of DCIS, in particular, is low (55%).\n16. Results: Results are reported at a fixed threshold (0.1); different thresholds are not explored.\n17. Discussion: Please consider highlighting the key findings from the study (including summary data) in the first paragraph.\n18. Discussion: Architectural distortion is routinely biopsied unless clearly associated with a known postsurgical scar.\nHowever, the pathology often reveals a radial scar, a benign finding. This contributes to the high false-positive rate\nassociated with distortion.\n19. Discussion: Please note that a lesion-specific analysis was not conducted. While the model identified 31% of interval\ncancers as positive, this does not guarantee that the lesion annotations were in the correct location of the subsequently\ndiagnosed breast cancer.\n20. Discussion: It could also be added as a limitation that the study simplifies exam-level labels by focusing on the most\nsevere finding.\n21. References: Prior work on this specific algorithm (Lunit INSIGHT DBT) is not cited: Radiol Artif Intell 2024; 6(3):e230318.\n22. Figure 1: For the abnormal screening exams (31,379), why is there such a large number (10,310) with missing diagnostic\nexams?\n23. Figure 1: Why are there 14 BI-RADS 6 cases among the abnormal screening exams?\n24. Figure 1: Please include a footnote with abbreviations. To what does BLL refer?\n25. Figures 2/3: These figures aren’t clear – please provide more detailed legends.\n26. Figure 6: Please confirm that these images are correctly annotated (specifically, TP architectural distortion and FN\narchitectural distortion).\n27. Figures A5/A6: These figures include descriptors that are typically not used in screening mammography reports.\n28. Table 1: Please clarify the definition of “Screen Negative” – the subsequent screening exam would be performed in 12\nmonths or later.\n29. Table 2: What is the difference between a borderline lesion and a high-risk lesion?\n30. Tables 2/3: Was the finding (mass, asymmetry, distortion, or calcifications) based on the screening mammogram report?", "reviewer_2": "The authors presented an analysis of the performance of an AI tool for digital breast tomosynthesis (DBT). This topic is\nhighly relevant due to the global shortage of radiologists and the promising performance of various AI tools in imaging.\nPlease find below some comments and suggestions for your consideration.\nIt is understood that data were collected from screenings at different sites, were those different hospitals? Could you please\nclarify which types of screening programs were involved? Were they for the general population or specifically for higher-risk\nwomen? Additionally, some women under increased follow-up due to a higher risk for breast cancer?\"\nIt is acknowledged that interval cancers are challenging to deal with in screening studies, as some may not be detectable\nduring the screening visit. Generally, interval cancers tend to be larger and have a worse prognosis. For a recent study on\nthis topic, please see this article, https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2770959. Therefore, an AI\ncapable of preventing cancers that could become interval cancers would be highly beneficial. Of note, visible interval cancer\ncases cannot be distinguished from regular overlooked cases at the time of screening. The authors report that 31% of\ninterval cancers were correctly classified (lines 249-251). Could the authors please elaborate on the impact of this finding on\nthe interpretation of the results? Additionally, could the authors present an analysis where interval cancers are considered\nalongside other types of cancers and compare the results?\nSection 2.4.4. Could you please clarify if exams with missing or invalid BI-RADS and with follow-up information also were\nexcluded? If so, please explain.\nPlease describe the criteria used to classify an exam as ”invalid”.\nI am having difficulty reconciling the numbers in Figure 1 with the text. According to line 102, the model evaluated 202,758\nexams, and 58 failed to process. I could not find the 58 exams indicated in Figure 1, nor do I understand how the number\n'202,758' was derived. Could you revise the figure to make it clearer for the reader? Additionally, the meaning of the different\ncolors in Figure 1 is not clear.\nPlease explain what 'failure to process' means. Is this related to women with implants?\nCould you please comment on whether the presence of implants affect the performance of the model?\nI am curious whether this software can be installed on scanners from manufacturers other than Hologic. Could you speculate\non how this might affect the generalizability of the results to other manufacturers?\nThe FPR, TNR , FNT are dependent on the underlying cancer prevalence in the population. Could you please elaborate on\nthis point in the discussion section?\nIn the discussion, several statistical results are mentioned, and claims of statistical significance are made. For example, it is\nstated: 'We observed a slight, but statistically significant decrease in model performance for heterogeneously dense breasts\n(density C) (AUC: 0.90, 95% CI: 0.88-0.91), and a trend of decreased performance in extremely dense breasts (density D)\n(AUC: 0.87, 95% CI: 0.82-0.92).' However, the statistical section does not describe any significance tests or comparisons.\nPlease detail the tests conducted in the statistical section and explain their relevance. If the term 'statistical significance' is\nused, please refer to the protocol or SAP to show that the tests performed were pre-planned and preserved the family-wise\ntype I error rate at 5%.\nMinor comments:\nConsidering that this journal has a wide audience, it would be helpful to include two sentences explaining the BI-RADS\nscale for readers who may be unfamiliar with it.\nTable 1 is difficult to read because there are no extra rows between the text, and the explanatory text starts above the\ncorresponding labels. Could horizontal lines or other layout modifications be considered to improve readability?\nSection 2.5.2 Could you please briefly explain how the ”commercial operation point of Insight DBT was determined?\nI am curious why the authors refer to 'AUC, recall, and precision' as machine learning metrics. While these metrics are\nindeed used in machine learning, they were originally developed to assess the adequacy of classification models. Do the\nAUC, recall, and precision metrics used in this study differ from the standard classification metrics?\nI have some concerns about the statement in the introduction regarding the effect of screening mammography on reducing\nmorbidity and mortality (lines 6 - 8). The studies that informed the introduction of breast cancer screening programs are\noutdated, and both the equipment and treatment options have improved considerably since then. To my knowledge, the\nimpact of BC screenig programms on mortality reduction in the context of current standards of care is not fully understood. A\nrecent analysis of newer observational studies, suggested that the protective effect may not be attributable to the screening\nprogram itself https://www.jclinepi.com/article/S0895-4356(24)00181-1/fulltext. I encourage the authors to revise this\nsentence in the introduction to provide a more updated view of the current evidence.\nThank you for sharing your work. I hope that you find my review helpful.\nKind regards,\nAldana Rosso"}, "author_response": "Response: Thank you for this comment. This has been mentioned in the limitations\nsection. Given that our exams are ComboHD (both 2D and 3D), unfortunately it is not\npossible to determine whether the radiologist detected calcifications on 2D or DBT. We\nhope to do a future evaluation that includes combined evaluation of the 2D and DBT model,\nhowever this was not possible for the current study.\nResponse: As suggested by the Reviewer, we have addressed this discrepancy in the\ndiscussion section and cited the article. Lines 406-408\nResponse: As suggested by the reviewer, the term DCIS has been used in place of non-\ninvasive cancer in the manuscript.\nResponse: We evaluated model performance at the exam level rather than the lesion level.\nAlthough our dataset includes lesion-level findings (e.g., in a dataframe) and separate\nRegion of Interest (ROI) annotations, there is no direct mapping between them. This\nbecomes a limitation when multiple lesions are present in a single exam, making it unclear\nwhich annotation corresponds to which finding. Accurate lesion-level analysis would\nrequire expert mapping, which we plan to address in a future study with the help of\nradiologists.\nResponse: Thank you for pointing this out. We investigated the large number of abnormal\nscreening exams without diagnostic follow-up and found that many had addendums.\nWe extracted the structured addendum data and updated the original BIRADS 0 assessments\n(often assigned due to the need for prior mammograms by the Radiologist for comparison\npurposes) to BIRADS 1 or 2 and updated BIRADS descriptors. Some of the existing exams\nwere updated as well if an addendum existed for them. We incorporated this information\ninto our dataset, which reduced the number of exams missing diagnostic follow-up from\n10,310 to 4,001 and increased the total number of exams included in the evaluation. As a\nresult, the numbers in our tables and figures have been updated, but the overall trends\nremain unchanged.\nResponse: Thank you for the suggestion. We have addressed this limitation in the\ndiscussion section. Lines 507-511\nResponse: The abstract has been updated to read “screen negatives, diagnostic negatives\nand biopsy-proven benign cases” to clarify the “various negative exam types”. We thank\nthe reviewer for the suggestion.\nResponse: Yes “non-invasive cancers” refer to ductal carcinoma in situ (DCIS). We have\nupdated the manuscript to reflect DCIS in place of “non-invasive cancers”. LCIS was not\ncounted as a non-invasive cancer for this study.\nResponse: Thank you for the suggestion. We have updated the manuscript to report\nsensitivity and specificity separately under section “1.1 Breast Cancer Screening and\nDigital Breast Tomosynthesis”. Lines 26 - 32\nResponse: Thank you for pointing this out. Based on the reviewer’s comment an update\nhas been made in the Background section to replace “three dimensional imaging” with\n“pseudo three dimensional imaging”. Line 36\nResponse: We have updated section “2.2 Data” to reflect this. Thank you for pointing it\nout. Lines 124-125\nResponse: As suggested by the Reviewer we have included under section “2.2 Data” to\nclarify 2D only exams are excluded from the analysis. Lines 125-126\nResponse: Yes “non-invasive cancers” refer to DCIS. The definition of high-risk lesions and\nborderline lesions is based on an ontology established by an expert panel of breast\npathologists, oncologists, and radiologists as described in the original EMBED manuscript.\nThe High Risk Lesion category includes: Atypical Ductal Hyperplasia (ADH), Flat Epithelial\nAtypia (FEA), Radial Scar, Lobular Carcinoma in Situ, Atypical Lobular Hyperplasia,\nHyperplasia with Atypia, Columnar Cell Change with Atypia, Intraductal Papilloma, Atypical\nPapilloma, and Microscopic Papilloma.\nThe Borderline Lesion category includes: Mucocele, Granular Cell Tumor, and Atypical\nPhyllodes.\nResponse: The sentence has been revised to reflect “Lastly, pathological subgroups were\ndefined based on the final diagnosis determined from biopsy or, when available,\nsubsequent surgical resection.” Lines 241-242\nResponse: Thank you for pointing this out. We agree this is lower than expected. The\nsensitivity for invasive cancers is 82%, but the overall sensitivity for screen-detected\ncancers is driven lower by poor performance on DCIS cases which tend to be\ncalcifications. This is discussed in the limitations as many of these calcifications may be\nbetter detected on 2D imaging. Lines 421-426\nResponse: The purpose of our study was to evaluate an FDA-cleared commercial model\nwhich operates at a fixed threshold. This threshold is not adjustable by the end user;\ntherefore, our results reflect anticipated real-world performance if this model was\ndeployed clinically.\nResponse: Thank you for this suggestion. We have moved the key findings to the first\nparagraph and moved some of the commentary to the conclusion.\nResponse: Thank you for the insight. We have addressed this in the discussion section.\nLines 429-432\nResponse: We included wording to address this in the limitations. Thank you for the\ncomment. Lines 502-506\nResponse: Thank you for the suggestion. We have updated the “Limitations” section to\nreflect this. Lines 502-504\nResponse: Thank you for bringing this to our attention. We have cited the work in the\n“Discussion” section. Lines 406-408\nResponse: Thank you for pointing this out. The response to Question 5 on addendums\naddresses this question as well.\nResponse: Thank you for this observation. These likely represent cases where a patient\nhad an abnormal screening at our institution followed by a diagnostic exam and biopsy\nelsewhere, before returning to our institution for further diagnostic workup. These were\nconsidered invalid exams and excluded earlier in the data handling to clarify this.\nResponse: Thank you for highlighting this. A note has been added to figure 1’s caption with\ndefinitions for HRL (High-Risk Lesion) and BLL (Borderline Lesion).\nResponse: The caption for Figures 2 and 3 has been revised for clarity. We have added text\nto specify what the red hashmark and faint dotted line represent. To maintain visual\nsimplicity and prevent clutter, we chose to include this information in the caption rather\nthan embedding a legend directly in the figure.\nResponse: We reviewed the images with a Breast Radiologist, and we selected new\nimages which project the lesion types better for the TP and FN architectural distortion and\nupdated Figure 6.\nResponse: Thank you for this comment. After further review and comparison with the\nBIRADS manual we identified the only atypical descriptor as ‘clustered’. It appears that\nclustered was last used in the BIRADS 4 manual from 2003 and may still be used by some\nradiologists even after release of the BIRADS 5 manual in 2013. We have merged clustered\nwith grouped calcifications since this is now the preferred term in BIRADS 5 manual. The\nother descriptors are confirmed to be used on screening mammograms.\nResponse: Thank you for the observation. We have revised to say “after 12 months” rather\nthan “within 12 months”.\nResponse: This definition can be controversial. It is typically agreed upon by experts that\nhigh-risk lesions are resected due to a high correlation with malignancy. Borderline lesions\nrepresent a unique class of controversial results in which some oncologists may choose to\ntreat these more aggressively whereas others may not. Overall BLL are rare, and typically\nnot resected. Our definition of HRL and BLL is referenced in the original EMBED paper and\nhas been added as Table A2 in this manuscript for clarity.\nThe High Risk Lesion category includes: Atypical Ductal Hyperplasia (ADH), Flat Epithelial\nAtypia (FEA), Radial Scar, Lobular Carcinoma in Situ, Atypical Lobular Hyperplasia,\nHyperplasia with Atypia, Columnar Cell Change with Atypia, Intraductal Papilloma, Atypical\nPapilloma, and Microscopic Papilloma.\nThe Borderline Lesion category includes: Mucocele, Granular Cell Tumor, and Atypical\nPhyllodes.\nResponse: Yes, these were based on descriptions entered by the radiologist during the\nscreening mammogram. This is a known limitation as the imaging descriptors can change\nbetween a screening and subsequent diagnostic exam, however reconciling the imaging\nfeatures on diagnostic exam with features on screening study is challenging to do\nautomatically at scale and would necessitate manual review of each case.\nResponse: Yes, these were four different hospitals.\nResponse: Thank you for the question. Our screening population contained women at both\naverage and above-average risk of breast cancer, as the AI model is cleared for all\nscreening exams. Risk information is currently being curated in EMBED so we may further\ndifferentiate in model performance between average and above-average risk women in\nfuture studies. We included wording to the “2.2 Data” section to address this. Lines 119-\nin breast cancer prediction models for the exact reason described. It is very difficult to\nreliably conclude if an interval cancer was indeed visible on the screening exam and\nmissed by the radiologist, or whether it was not detectable. Given the small number of\nsamples, it is difficult to draw conclusions on the performance of any AI model for interval\ncancers. We have added this comment to the discussion. Lines 478 - 484\nResponse: Thank you for this question. Screening exams with missing or invalid BI-RADS\nwere excluded. If they had any follow-up information too, they were excluded.\nResponse: Thank you for your feedback, we’ve expanded section 2.4.4 (Excluded/Invalid\nExams) of the Methods to discuss criteria for excluding exams in more detail. Lines 209-\nrevised value is 229,217 exams were evaluated by the Lunit DBT Insight model with 1 exam\nfailing to process. Excluding the single failed exam, model outputs were retrieved for\n229,216 exams. Of these, 9 exams had invalid de-identified accession numbers due to\nissues during the data anonymization process, making it impossible to match them to the\nground truth data. This can occur when the original accession numbers are in an unusual\nformat, likely due to rare data entry errors. These 9 exams were excluded, leaving 229,207\nexams for evaluation. 202,923 of these corresponded to screening exams and 26,284\ncorresponded to diagnostic exams that were excluded from consideration.\nThe numbers have also increased in the revised manuscript. Reviewer 1 pointed out on the\nabnormal large number of abnormal screening exams that do not have a diagnostic follow-\nup. We looked into this discrepancy and found that many of these cases had addendums\nwhich were not being considered. The structured addendum data was extracted, and it\nupdated the original BIRADS 0 assessments (often assigned due to the need for prior\nmammograms by the Radiologist for comparison purposes) to BIRADS 1 or 2 and updated\nBIRADS descriptors. Some of the existing exams were updated as well if an addendum\nexisted for them. This information was incorporated into our dataset, which reduced the\nnumber of exams missing diagnostic follow-up from 10,310 to 4,001 and increased the total\nnumber of exams included in the evaluation. As a result, the numbers in our tables and\nfigures have been updated, but the overall trends remain unchanged.\nResponse: Here “failing to process” means images that failed to be inferenced by the\nmodel due to issues while reading or transforming the image data, specifically stemming\nfrom missing or malformed DICOM information. Thank you for the question. We have also\nexpanded on this in the “2.2 Data” section. Lines 131-133\nPerformance was not specifically evaluated in women with breast implants as the EMBED\ndata right now does not reliability identify these patients. The “4.1 Limitations” section is\nupdated to reflect this comment. Lines 511-513\nResponse: The software is currently FDA-approved for use with Hologic and GE devices.\nWhile the underlying AI algorithm may be compatible with images from other\nmanufacturers in theory, its performance may vary due to differences in image acquisition\nprotocols, resolution, and preprocessing. The same level of accuracy or generalizability for\nother vendors currently cannot be guaranteed.\nResponse: Thank you for pointing it out. We have added the suggested content under “4.1\nLimitations” and expanded to say, Emory as a tertiary referral center has slightly higher\nprevalence of cancer cases compared to the general population, which can affect\nperformance metrics such as the false positive rate (FPR), true negative rate (TNR), and\nfalse negative rate (FNR), as these metrics are influenced by disease prevalence. Lines 491-\nEvaluation” describing the statistical methods. We have clarified the statistical methods\nused to support claims of significance in the revised manuscript. Comparisons between\nsubgroups were conducted using Kruskal-Wallis test on bootstrapped AUC distributions,\nfollowed by post-hoc pairwise comparisons where appropriate. The Bonferroni correction\nwas applied to control the family-wise type 1 error rate at 5%. Results were described as\n“statistically significant” only if they met the corrected threshold. Lines 268-282\nResponse: Thank you for the suggestion. We have added text explaining the BI-RADS scale\nunder “1.1 Breast Cancer Screening and Digital Breast Tomosynthesis”. Lines 11-22\nResponse: Thank you for the suggestion. We have formatted the table for better readability\nby adding space between each row.\nResponse: The threshold was selected based on evaluation across various operating\npoints to balance sensitivity and specificity for cancer detection. It offered an optimal\nsensitivity of 0.9 while maintaining an acceptable false positive rate aligning with the\nproduct’s intended use. We have included this explanation under “2.1 Model”. Lines 110-\nDiscussion” to reflect “classification metrics” in place of “machine learning metrics”. Line\nWe have revised the introduction to include this context and relevant citations. Lines 6-10"}, {"round": 2, "reviewer_comments": {"reviewer_1": "Summary\nIn this retrospective validation study, the authors evaluate the performance of a commercial AI-based CAD tool (Lunit\nINSIGHT DBT) for breast cancer detection on DBT. The analysis includes more than 167,000 screening mammograms from\nthe Emory Breast Imaging Dataset and examines model performance across demographic, pathologic, and imaging\nsubgroups. The model attained an overall AUC of 0.91. However, significantly lower performance was observed in certain\nsubgroups, including cases with DCIS, calcifications, and dense breast tissue. The authors conclude that the model\ndemonstrates strong overall performance but that careful consideration of subgroup-specific limitations is essential for its\nclinical implementation.\nStrengths\nA. There is limited research on the performance of AI-based CAD models for breast cancer detection across demographic,\npathologic, and imaging subtypes. Most studies focus solely on overall performance metrics without addressing subgroup-\nspecific variations.\nB. A strength of the study is its detailed failure analysis, which offers valuable insights into the model's limitations and error\npatterns.\nC. The analysis includes more than 167,000 screening exams from a diverse population.\nWeaknesses\nNone noted. All prior comments have been appropriately addressed.", "reviewer_2": "Dear Authors,\nThank you for addressing my questions and considering my suggestions.\nI have no further comments.\nKind regards,\nAldana Rosso"}, "author_response": ""}]

Decision letter