An interpretable AI system reduces false-positive MRI diagnoses by stratifying high-risk breast lesions

Domain: neuroscience
Published: 2026-02-02
Editorial outcome: major_revision

Abstract
Breast cancer diagnosis using magnetic resonance imaging remains limited by high false-positive rates and substantial inter-reader variability, especially for lesions classified as Breast Imaging Reporting and Data System (BI-RADS) category 4, often leading to unnecessary biopsies. Here we show that the BI-RADS 4 Lesions Analysis System (BL4AS), an artificial intelligence system powered by foundation models and leveraging the rich spatiotemporal information of dynamic contrast-enhanced MRI, addresses these diagnostic challenges. Developed on a multicenter dataset of 2,803 lesions from 2,686 female patients, BL4AS demonstrates robust performance with areas under the curve of 0.892-0.930 and significantly outperforms radiologists in specificity (0.889 versus 0.491). BL4AS-assisted interpretation significantly improves diagnostic accuracy for both senior and junior radiologists, reducing inter-reader variability by 24.5% and decreasing false-positive rates by 27.3%. BL4AS further stratifies lesions into subcategories (4A, 4B and 4C) for refined risk assessment, offering a practical tool for precision breast cancer management.

Introduction
Breast cancer, with over 2.3 million new cases diagnosed annually, remains a leading cause of cancer-related mortality worldwide1,2. Its high incidence and significant mortality underscore the critical need for early and accurate detection. Over the past two decades, breast magnetic resonance imaging (MRI) has become an indispensable tool, particularly for high-risk populations3–7. While MRI significantly enhances sensitivity compared to mammography or ultrasound imaging techniques, its specificity remains suboptimal, leading to unnecessary biopsies and follow-up procedures8,9. This challenge is particularly acute for Breast Imaging Reporting and Data System (BI-RADS) category 4 lesions10, which encompass a wide range of malignancy probabilities (2% to 95%)11, and currently require biopsy for definitive diagnosis12. Thus, developing methods to stratify BI-RADS 4 lesions and reduce unnecessary biopsies is a pressing clinical demand.

The interpretation of breast MRI is highly subjective, with significant intra- and inter-observer variability13,14. Less experienced radiologists are more likely to misclassify benign lesions as suspicious, resulting in unnecessary biopsies, while subtle malignant lesions may be overlooked, leading to delayed diagnoses15. Moreover, the current diagnostic workflow is resource-intensive, requiring specialized training and time, which limits its scalability16. The lack of standardized criteria for BI-RADS 4 lesions further exacerbates these issues, as the category has a broad spectrum of malignancy probabilities, making it difficult to balance sensitivity and specificity. For instance, it has been reported that the positive predictive value (PPV) of MRI-guided biopsies is only 20.5–30.3%17–19. These limitations underscore the need for innovative solutions to improve diagnostic accuracy, consistency, and efficiency.

Recent advances in deep learning have shown promise in breast cancer detection across imaging modalities20–23. However, existing AI systems face critical limitations, including lack of robust external validation, limited prospective evaluation, and insufficient interpretability, which hinder clinical adoption24–28. To address these gaps, we developed the BI-RADS 4 Lesion Analysis System (BL4AS), a foundation model-powered AI system29–32 that leverages the rich spatiotemporal information of dynamic contrast-enhanced MRI (DCE-MRI). Unlike previous studies that rely on single images or maximum intensity projection images33,34, BL4AS integrates multiphase imaging (peak, inflow, and outflow phases) to capture comprehensive lesion dynamics, enabling more accurate characterization.

In this work, we present the BL4AS, a foundation model-powered artificial intelligence system that leverages spatiotemporal information from multiphase DCE-MRI to enhance diagnostic accuracy and reduce unnecessary biopsies for BI-RADS category 4 lesions. Rigorously validated on multicenter external and prospective test sets, BL4AS demonstrates robust generalizability and outperforms radiologists, particularly in specificity and negative predictive value (NPV). A reader study involving both senior and junior radiologists demonstrates BL4AS’s ability to support clinical decision-making. Additionally, interpretable visualizations of model focus areas foster clinician trust and facilitate adoption. By improving diagnostic consistency and efficiency, BL4AS has the potential to significantly enhance patient outcomes and optimize resource utilization in breast cancer care.

Results
Studydesign
The overview of this work is illustrated in Fig. 1. We first constructed a multicenter breast MRI dataset to develop and evaluate our BL4AS model. In total, 2686 patients with 2803 lesions were collected from multiple clinical centers, incorporating the rich spatiotemporal information of DCE-MRI. The study specifically focused on BI-RADS 4 lesions with complete preoperative MRI examination data and pathologically confirmed diagnoses. A total of 2686 retrospectively collected female patients (median age, 47 years) with 2803 lesions were included in this multicenter study. Training and validation sets comprised 1881 patients (582 benign, 1,355 malignant), while external tests A and B included 103 patients (61 benign, 47 malignant) and 614 patients (513 benign, 153 malignant), respectively. Additionally, a prospective test set of 88 participants (median age, 51 years) were recruited, with 27 (29.3%) benign and 65 (70.7%) malignant lesions. Baseline characteristics of the patients across all datasets are summarized in Table 1, while detailed comparisons between retrospective and prospective datasets are presented in Supplementary Table 1.

We trained and assessed the diagnostic performance of our BL4AS model against existing AI methods using retrospective data and conducted ablation studies to validate model components. Subsequently, eight readers with varying experience provided independent diagnostic assessments in the prospective test set, then re-evaluated the same cases with BL4AS assistance to validate clinical effectiveness. Finally, we enhanced BL4AS interpretability through Grad-CAM visualizations, which highlighted the key regions that the model focused on during decision-making, and compared these attention patterns to clinical diagnostic preferences. Additionally, we evaluated BL4AS’s potential for optimizing clinical management strategies through refined risk stratification of BI-RADS 4 lesions.

Models’performanceonretrospectivedata
BL4AS demonstrated robust performance in lesion segmentation, achieving Dice similarity coefficients of 0.851, 0.834, 0.825 and 0.837 in the training, validation and external test sets, respectively (Supplementary Table 2). For lesion classification, we conducted three ablation experiments: (1) Pretraining Impact: Vision Transformer (ViT) models with the pre-trained foundation models outperformed non-pretrained models across all metrics (Fig. 2a); (2) Multiphase Fusion: ViT models integrating multiphase DCE images (Cpeak, Csub1, Csub2) performed significantly better than single-phase models (Fig. 2b–d); (3) Feature Integration: Among various feature integration methods, our image-based fusion approach (ImageConv) performed better than other methods in most scenarios (Fig. 2e–g).

To establish the superiority of our approach, we compared BL4AS against multiple baseline methods adapted for breast lesion diagnosis. These included traditional machine learning algorithms utilizing handcrafted radiomics features, as well as state-of-the-art deep learning architecture. Our results demonstrated that BL4AS outperformed other methods across the majority of evaluation metrics (Supplementary Data 1), with particularly notable improvements in specificity for BI-RADS 4 lesions. Ultimately, BL4AS achieved AUC values of 0.896 (95% Confidence Interval [CI]: 0.832–0.949) and 0.930 (95% CI: 0.906–0.950) in external test sets A and B, respectively (Fig. 3a, Table 2).

To evaluate model robustness and generalizability across diverse clinical scenarios, we performed subgroup analyses based on patient demographics, lesion characteristics, and imaging protocols. The findings indicated that BL4AS exhibited satisfactory predictive performance across combined validation and external test sets when evaluated across different patient age groups, various lesion characteristics including size, enhancement patterns, and background parenchymal enhancement (BPE) levels, and diverse imaging protocols (1.5T/3T, different scanner types), as shown in Supplementary Fig. 1.

Prospectivevalidationandreadercomparison
To evaluate real-world applicability, BL4AS was tested on a prospective test set from Center 1, achieving an AUC of 0.892 (95% CI: 0.813–0.951; Fig. 3b). Compared to eight readers, BL4AS demonstrated significantly higher specificity (0.889 vs. 0.333–0.741, P < 0.05 for seven of the eight readers; Fig. 3b, Supplementary Data 2). Supplementary Data 2 summarizes the sensitivity and specificity results of BL4AS and individual readers. The sensitivity of readers ranged from 0.785 to 0.908 and the specificity ranged from 0.333 to 0.741, while BL4AS maintained a balanced performance with sensitivity of 0.862 (95% CI: 0.773–0.944) and specificity of 0.889 (95% CI: 0.750–1.000). BL4AS also outperformed both junior (AUC: 0.635) and senior (AUC: 0.729) readers (Fig. 3b, Table 3). Additionally, BL4AS completed lesion classification in approximately 2.03 seconds, significantly faster than the average reader time of 137 s.

Human-BL4ASinteractioninclinicpractice
To address BL4AS’s clinical utility, the same eight readers reviewed the same cases with BL4AS assistance. As shown in Table 3 and Fig. 4a and b, BL4AS improved the sensitivity of all readers by 3.7% and the specificity by 27.3%, particularly significant for junior R5, whose specificity increased from 0.333 to 0.815 (Supplementary Data 2). Moreover, BL4AS boosted the readers’ diagnostic accuracy with a lower false positive rate and a lower false negative rate (Fig. 4c and d). The false positive rate significantly decreased from 50.9% (range: 23.0–58.7%) to 23.6% (range: 6.9–38.5%) in the prospective test set (P=0.012), potentially reducing unnecessary biopsies (Fig. 4d). Reader agreement also improved, with the average kappa value increasing from 0.501 (range: 0.244–0.702) (Fig. 4e) to 0.746 (range: 0.662–0.852) (Fig. 4f) in the prospective test set. We further evaluated the differential effects of AI assistance across various clinical scenarios using generalized estimating equation models to assess interaction effects between AI assistance and subgroup variables. As shown in Supplementary Fig. 2, the analysis revealed interactions for sensitivity regarding age group (P=0.015), and for specificity regarding lesion size (P=0.010), enhancement pattern (P < 0.001), magnetic field strength (P=0.002), and scanner type (P=0.026), indicating that the magnitude of AI assistance benefits varied across different patient characteristics and imaging parameters. Despite these differential effects, BL4AS consistently improved reader performance across most evaluated subgroups, including varying BPE, age, lesion size, magnetic field strength, scanner type, and lesion enhancement pattern (Fig. 4g, Supplementary Fig. 3). Notably, in cases with moderate or marked BPE where readers often face assessment challenges, BL4AS assistance substantially improved readers’ diagnostic performance compared to their independent evaluations (Fig. 4g). Moreover, with BL4AS assistance, readers completed lesion assessments in approximately 68 seconds, half the time required without AI support.

Interpretationandvisualization
To enhance clinical credibility, BL4AS incorporates heatmaps highlighting regions critical to its predictions (Fig. 5, Supplementary Fig. 4). These visualizations aided readers in refining diagnoses by identifying key imaging features, such as irregular margins and abnormal enhancement patterns. However, some misclassifications occurred, particularly for lesions with peripheral enhancement or overlapping benign-malignant features (Supplementary Fig. 4). Our detailed analysis of false negative cases revealed that these misclassifications predominantly involved small lesions, circumscribed margins, and BI-RADS 4a subcategory (Supplementary Table 3). Pathologically, BL4AS misclassified ductal carcinoma in situ and mucinous carcinoma cases more frequently, likely due to their less invasive imaging characteristics that closely mimic benign lesions. Notably, incorporating additional sequences particularly DWI showed potential to assist in better characterizing these morphologically ambiguous cases (Supplementary Figs. 5–8).

Clinicalmanagementstrategies
BL4AS demonstrated potential to optimize biopsy decision-making. Alternative hypothetical operating points could be selected based on specific clinical situations (Fig. 6a). At a high NPV operating point, BL4AS prevented 123 unnecessary biopsies per 1000 patients while maintaining 100% sensitivity in external test set B. Conversely, at a high-PPV operating point, BL4AS ensured prompt intervention for high-risk lesions.

Furthermore, BL4AS could stratify BI-RADS 4 lesions into subcategories (4A, 4B and 4C) with high accuracy (Fig. 6b). In external test set B, malignancy rates were 2.2%, 35.6%, and 86.2% for 4A, 4B, and 4C, respectively. Similarly, external test set A showed corresponding rates of 9.8%, 44.1%, and 84.8%. BL4AS identified more accurately low-risk lesions (4A malignancy probability: 2.2% vs. 9.2%) and reduced false negative rates (6.1% vs. 30.0%) for high-risk categories (4B/4C) compared to radiologists (Fig. 6c and d). This improved risk stratification was reflected in a significant Net Reclassification Improvement (NRI) of 17.45% (95% CI: 9.15–25.75%, P < 0.001), highlighting the potential of BL4AS to individualize follow-up strategies and optimize resource allocation.

Methods
Ethicalapproval
This study was approved by the institutional review board of all participating hospitals. For the retrospective analysis, a waiver of informed consent was granted due to the use of de-identified data. For the prospective study, written informed consent was obtained from all participants (Clinical Trial Registration: ChiCTR2400081831). The study adhered to the Checklist for Artificial Intelligence in Medical Imaging (CLAIM)50.

Studydesignandparticipants
We retrospectively collected data from female patients who underwent a breast MRI in accordance with current recommendations51 at three academic medical centers, including Guangdong Provincial People’s Hospital (Center 1), Peking University Shenzhen Hospital (Center 2) and Guangdong Women and Children’s Hospital and Health Institute (Center 3). All data were anonymized and deidentified, as per the Health Insurance Portability and Accountability Act Safe Harbor provision52. Indications for MRI included evaluation of suspicious clinical abnormalities, equivocal findings on mammography and/or ultrasound, high risk screening, and assessment of the extent of a known breast cancer. The inclusion criteria were as follows: presence of an enhanced lesion classified as BI-RADS 4 on MRI; pathology confirmed breast lesions following biopsy or surgery. The exclusion criteria were as follows: any chemotherapy, radiotherapy, or hormone treatment was performed before MRI; poor imaging quality, which compromised lesion observation and delineation; mismatch between MRI and the location of biopsy or surgically identified lesions. For multifocal lesions, the largest lesion was selected for analysis. Patients from Center 1 were randomly divided into training (80%) and internal validation (20%) sets using stratified sampling to maintain balanced representation of malignant and benign cases across both subsets. Patients from Center 2 and Center 3 formed independent external test sets A and B to assess cross-institutional generalizability. All partitions were disjoint at the patient level, ensuring that images from the same patient appeared in only one dataset. To further evaluate the model performance and compare it with clinical assessments, we prospectively recruited patients from Center 1 for the prospective test set. Sex was determined based on the government-issued IDs. Details of the inclusion and exclusion criteria are provided in Supplementary Fig. 9.

MRIacquisitionanddatapreprocessing
Breast MRI was performed using 1.5T or 3T scanners with dedicated breast coils in the prone position. Imaging protocols followed international guidelines and included T2WI, DWI and DCE images acquired before and after gadolinium-based contrast agent injection. Detailed acquisition parameters are provided in Supplement Methods and Supplementary Table 4.

Standardized preprocessing pipeline to ensure consistency across multicenter data while accommodating protocol variations, a comprehensive standardized preprocessing pipeline was implemented using identical software tools and parameters across all centers: (1) Alignment and Registration: Spatial alignment of DCE-MRI sequences using the open‑source deedsBCV package in its default setting. (3) N4 Bias Field Correction: Intensity inhomogeneity correction using SimpleITK’s N4BiasFieldCorrectionImageFilter in its default setting. (4) Resampling: Standardization to 1×1×1 mm³ isotropic voxel spacing. Gray‑scale data employed third‑order B‑spline interpolation and binary masks were resampled with nearest‑neighbor interpolation to avoid label mixing. (5) Intensity Normalization: All intensity values were normalized using the mean and s.d. calculated from the training set to minimize inter-scanner intensity discrepancies.

Modeldevelopmentandtraining
BL4AS is a deep learning system specifically designed to perform two primary tasks: lesion segmentation and lesion classification (benign vs. malignant). Foundation models are large-scale, pre-trained neural networks trained on diverse datasets that can be adapted for various downstream tasks through fine-tuning. To enhance generalizability and performance, both tasks utilized a pretrained foundation model as their architectural foundation. The backbone of BL4AS is a Vision Transformer (ViT)53 architecture initialized with weights from a foundation model pretrained on 17,149 volumes (2.5 million DCE-MRI slices) using the Momentum Contrast (MoCo)-V3 self-supervised learning algorithm54,55. This pretraining enables the model to capture robust feature representations from DCE-MRI data, improving performance on downstream tasks.

For lesion segmentation task, we implemented a UNETR model architecture that utilizes the pretrained foundation model backbone. The model accepts input volumes with dimensions of 1×48×48×48 and generates corresponding output volumes of 2×48×48×48, where the two channels represent background and tumor regions respectively (Supplementary Fig. 10). Given that the original medical images typically exceed the 48×48×48 input size constraint of the model, we employed a sliding window approach for segmentation. This technique enables processing of larger volumetric data by dividing them into overlapping patches of the required input size, performing predictions on each patch, and subsequently aggregating the results to reconstruct the complete segmentation map of the entire volume. Additional implementation details are provided in the Supplementary Methods. The predicted tumor regions were cropped using bounding boxes. To ensure segmentation accuracy across all datasets, an experienced radiologist reviewed and manually corrected all model-generated segmentations (both training and test sets) when necessary before cropping the volumes of interest. The correct volumes of interest were subsequently used for further classification analysis.

For lesion classification task, the foundation model is extended with a multi-temporal fusion module and a patch-based discrimination module (Supplementary Fig. 11). The multi-temporal fusion module captures lesion enhancement patterns by integrating three phases of DCE-MRI: Cpeak (images with peak contrast enhancement), Csub1 (subtraction images between Cpeak and pre-contrast phase images), and Csub2 (subtraction images between Cpeak and delayed post-contrast images). These temporal phases are integrated through 3D convolutional blocks to generate spatiotemporal representations. The patch-based discrimination module enhances the ViT architecture by generating lesion-specific tokens, including a classification token that captures global contextual information and a series of patch tokens that preserve local spatial details for fine-grained discrimination. Lesion masks were used to differentiate between lesion and background regions, reducing background interference and improving focus on lesion features.

The model is trained using the AdamW optimizer with a learning rate of 1e-4, and the loss function incorporated Dice loss for segmentation and cross-entropy loss for classification. All experiments were conducted on a single NVIDIA RTX 3090 GPU, with detailed hyperparameters and implementation provided in the Supplement Methods.

Comparativeanalysiswithexistingmethods
While previous studies have proposed various AI models for breast lesion classification, none specifically address BI-RADS 4 lesion malignancy prediction. To provide comprehensive performance benchmarks, we implemented multiple baseline approaches, including traditional machine learning with handcrafted radiomics features and state-of-the-art deep learning architectures. All models were trained and evaluated using identical datasets, preprocessing pipelines, and protocols for fair comparison. Detailed training parameters, hyperparameter settings, and implementation details for all comparison methods are provided at GitHub (https://github.com/zhenweishi/BL4AS).

Subgroupanalyses
To evaluate the robustness of our findings across different clinical scenarios and technical parameters, we conducted subgroup analyses in the combined validation and external test sets, including age (≤50 years and >50 years), lesion size (≤2 cm and >2 cm), lesion enhancement patterns (mass and non-mass), BPE levels (minimal or mild and moderate or marked), magnetic field strength (1.5T and 3T) and scanner types (GE, Philips and SIEMENS).

Readerstudy
Eight readers independently reviewed the images in the prospective test sets, including four senior readers (R1-R4: 5–15 years of experience) and four junior readers (R5-R8: 2–4 years of experience) in breast imaging interpretation. All readers were blinded to each other, to the original interpretation and to the pathological results. When readers made the judgments of benign and malignant classifications, reading time was also recorded. The performance of BL4AS was compared to visual assessment by readers using the maximum Youden index of the training set as the cutoff value.

AI-assistedreaderstudy
To assess the utility of BL4AS in aiding radiologists’ clinical decision-making, an AI-assisted reader study with a fully crossed design was conducted. The same 92 breast lesions in the prospective test set were presented again to the same eight readers 4 weeks after the end of the first assessment. For each lesion, together with the original MRI images, the corresponding AI prediction on malignancy risk probability were provided to the readers at the same time. Besides, all readers were blinded to the AI model’s performance in order to reduce expectation bias and ensure an unbiased assessment. Additionally, Cohen’s kappa value was used to calculate the agreement between all pairs of readers with and without BL4AS assistance. Additionally, we analyzed the performance of BL4AS and readers across various subgroups, including magnetic field strength, MRI scanner type, age, lesion size, the degree of BPE and lesion enhancement pattern.

BL4ASinterpretationandvisualization
To improve model interpretability, the Grad-CAM approach was employed to investigate and visualize which areas in the image are important for prediction. By calculating the weighted sum of the gradients flowing into the last convolutional layer of the network, Grad-CAM generates a heatmap that highlights the image regions most relevant to the prediction. Following the initial reader assessment, controversial cases with discordant interpretations were identified. It was interesting to explore the performance of BL4AS in these cases. Furthermore, to better understand the performance of BL4AS, we focused on the cases where BL4AS made prediction errors. Given the critical importance of false negative cases in oncologic diagnosis, we conducted a comprehensive analysis to explore the imaging and pathological features of these misclassifications.

Clinicalmanagementandriskstratificationanalysis
We evaluated BL4AS performance using dual operating point strategy optimized for different clinical priorities. The high-NPV threshold was designed to maintain 100% sensitivity while maximizing identification of truly benign lesions potentially suitable for surveillance. The high-PPV threshold was optimized to minimize false positive predictions, thereby ensuring that lesions flagged as high-risk truly require immediate intervention.

Additionally, we conducted risk stratification analysis to evaluate whether BL4AS could effectively subclassify BI-RADS 4 lesions (4A, 4B, and 4C) according to the ACR BI-RADS guideline. We assessed the actual malignancy rates in each subcategory across both external test sets. To assess the potential clinical impact, we compared the performance between BL4AS and radiologists’ assessments in external test set B. The NRI was calculated to quantify the improvement in risk stratification.

Statisticsandreproducibility
The performance of all models used for the classification tasks was evaluated in terms of AUC, accuracy, sensitivity, specificity, PPV and NPV. The primary outcome was the diagnostic performance of BL4AS for classifying BI-RADS 4 breast lesions, with secondary outcomes including (1) improvement in radiologists’ diagnostic performance and reading efficiency when assisted by BL4AS and (2) assessment of the model’s potential for clinical management and routine practice utility. The DeLong test56 was applied to compare differences between the AUC values. 95% CIs were calculated based on 1000 bootstrap replicates. Differences between the performance of BL4AS and readers with or without the BL4AS aid were calculated using McNemar’s χ2 test or Wilcoxon rank sum test. Continuous variables were compared by Kruskal-Wallis test, while qualitative variables were analyzed by two-sided Pearson’s Chi-square test or Fisher’s exact test. Sample size calculations are detailed in the Supplemental Methods. Interaction effects between AI assistance and subgroup variables were assessed using generalized estimating equation models. All experiments and analyses were performed following standardized protocols to ensure reproducibility. Inter-observer and intra-observer segmentation consistency were assessed and demonstrated robust annotation reproducibility. Statistical analyses were conducted using R (version 4.4.2) and Python (version 3.9). A two-sided P < 0.05 was considered statistically significant.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Dear sirs,\nYou have provided an interesting study on an AI algorithm for the classification for lesions in breast MRI.\nMy major issue is that I do not really understand why you limited yourselves to bi-rasd 4 lesions. It makes much more sense\nto classify all lesions in the breast and aim to prevent biopsy in a large proportion of these cases. You in essence show that\nthat is possible, but I cannot understand in how many cases biopsy could simply be avoided.\nThe classification into a/b/c is useful, but in practice does not lead to different approaches.\nIn addition, your algorithm also does segmentation, but I did not get clear what ground truth was used for that, or how you\nenvision using that. I.e. that seems to be a bit of a loose end.\nIn general, your results section is hard to understand without the methods section. It would be better to at least introduce a\nlittle the applied methods too, because of the manuscript design of nature comm.\nYour test-set is relatively small. That is not an issue, but when you move towards subset analysis, I expect that the\nconfidence intervals will be huge. that is not visible from fig 3. Please check.\nA few specific comments:\n67. The training and validation sets comprised 1,881 patients with\n68. 84 582 benign lesions, => whi didn’t you use malignant lesions in the training dataset?\n69. BL4AS demonstrated robust performance in lesion segmentation, achieving Dice similarity\n70. 91 coefficients of 0.851, 0.834, 0.825 and 0.837 in the training => you diod not even state that the source images were\nannotated, so how do you produce dice data. Can you elaborate on the annotation process?\n71. To evaluate real-world applicability, BL4AS was tested on a prospective test set from Center 1, 103 achieving an AUC of\n0.892 = > you seem to test only in patients with known lesions. Did you actually point out the lesions to be assessed or do\nyou do detection too?\n72. the same eight readers reviewed the same cases with BL4AS 114 assistance +> can you explain the design of the\nreader study? Is it sequential or fully crossed. Were readers informed of the AI quality?", "reviewer_2": "Manuscript#: NCOMMS-25-20370\nTitle: An Interpretable AI System Reduces False-positive MRI Diagnoses by Stratifying High-risk Breast Lesions\nThe manuscript presents BL4AS (BI-RADS 4 Lesions Analysis System), a deep learning-based AI model designed to\nstratify BI-RADS 4 breast lesions using dynamic contrast-enhanced MRI (DCE-MRI). Trained on a substantial dataset of\n17,149 MRI volumes, BL4AS demonstrates high diagnostic performance across multiple external and prospective test sets,\nachieving AUCs ranging from 0.892 to 0.930. Notably, the system significantly reduces false-positive rates and inter-reader\nvariability and enhances radiologists' specificity and efficiency when used as a decision-support tool.\nA prospective test with 88 cases demonstrated improved diagnostic consistency among radiologists when aided by the\nsystem. The model also provides interpretable Grad-CAM visualizations to support clinical decision-making. Despite\nencouraging results, the study's limitations include a small prospective sample and lack of formal adherence to reporting\nstandards like CLAIM 2 (Checklist for Artificial Intelligence in Medical Imaging) guidelines [Radiology, doi:\n10.1148/ryai.240300].\nSignificance and Comparison with Established Literature\nThis work addresses a relevant clinical problem in breast imaging: the overdiagnosis and low specificity of MRI in BI-RADS\n4 lesions. While similar efforts have been reported (e.g., radiomics or CNN-based classifiers for breast MRI), few have\nemployed foundation models with prospective testing. However, key works such as Daimiel Naranjo et al. (Cancers, 2022,\ndoi: 10.3390/cancers14071743) and Sun et al. (J. Transl. Med., 2021, doi: 10.1186/s12967-021-03117-5) have tackled\nsimilar goals.\nSupport for Conclusions and Need for Additional Evidence\nThe conclusions are generally supported by the results, particularly in terms of improved specificity and inter-reader\nagreement. However, the small prospective sample size (n=88) undermines the strength of the claims related to clinical\nimpact. Furthermore, there is no sample size or power justification in the statistical methods, which is essential for validating\nthe adequacy of the test cohort. A major omission is the lack of a detailed analysis of false negatives, which are critical in\noncologic diagnostics.\nFlaws in Data Analysis, Interpretation, and Conclusions\n- The limited prospective cohort size and single-institution origin introduce concerns of overfitting and limit generalizability.\n- Absence of formal sample size calculations and interaction testing in subgroup analysis weakens the methodological rigor.\n- Grad-CAM is a reasonable starting point but provides only limited insight into decision boundaries.\nAnother major limitation of the manuscript is the lack of direct comparison with existing AI models. While BL4AS\ndemonstrates strong performance within its own dataset, the absence of a quantitative comparison against previously\npublished models limits the ability to assess its true innovation and clinical added value.\nIn the methods section, two critical design choices are reported without sufficient justification and should be clarified by the\nauthors. First, the decision to select only the largest lesion in cases of multifocal disease may introduce bias, particularly if\nsmaller lesions have different imaging characteristics or histopathological profiles. The rationale for excluding additional\nlesions should be explained, along with its potential impact on model generalizability and performance. Second, the division\nof patients from Center 1 into training and internal test sets using an 8:2 ratio, and the assignment of patients from Centers 2\nand 3 to external test sets A and B, respectively, is described without justification. The authors should clarify whether\nstratification or balancing was applied during the split and explain why this center-specific allocation was chosen, especially\ngiven the possible influence of institutional imaging protocols and patient populations on model performance. Additionally,\nthese omissions result in the lack of compliance with CLAIM items 19–21, which require a detailed explanation of data\npartitioning, the rationale for dataset construction, and potential sources of bias or confounding.\nThe manuscript does not report whether any analysis was performed to assess differences in imaging data derived from\n1.5T versus 3T scanners. Given that image quality, signal-to-noise ratio, and enhancement characteristics can vary\nsignificantly between field strengths, this may influence both feature extraction and model performance. Could the authors\nclarify whether such a comparison was conducted, and if not, provide justification for its omission?\nData and Methodology\nThe prospective validation cohort includes only 88 cases, which is, in my opinion, insufficient to draw strong or\ngeneralizable clinical conclusions. This limited sample size significantly reduces the statistical power of the study and raises\nconcerns about potential selection bias. Moreover, the manuscript does not include any formal justification or power analysis\nto validate the adequacy of the sample size for prospective evaluation. Furthermore, the prospective cohort used for real-\nworld testing (Table 1) includes a disproportionate prevalence of malignant cases 65/88 (70.7%), which does not reflect the\ntrue distribution typically encountered in clinical practice for BI-RADS 4 lesions. This artificially high prevalence may bias\nperformance metrics, especially positive predictive value, and limits the generalizability of the findings.\nThe manuscript does not provide a comprehensive analysis of false negative cases, which is a critical omission. Given the\nhigh clinical risk associated with missed malignancies in breast imaging, detailed examination of these errors is essential.\nUnderstanding where and why the model fails would be necessary to assess its safety and clinical reliability.\nAnalytical Approach\nThe analytical approach is sound, employing appropriate performance metrics (AUC, specificity, sensitivity) and\ncomparative analyses with radiologists' assessments. To enhance this section:\n- Discuss the rationale behind chosen thresholds for classification and their clinical implications.\n- Consider performing subgroup analyses based on lesion characteristics or patient demographics to explore model\nperformance across different scenarios.\nThe manuscript does not provide a comprehensive analysis of false negative cases, which is a critical omission. Given the\nhigh clinical risk associated with missed malignancies in breast imaging, detailed examination of these errors is essential.\nUnderstanding where and why the model fails would be necessary to assess its safety and clinical reliability.", "reviewer_3": "Dear Colleagues,\nThe study presents an innovative and timely contribution to breast imaging by addressing one of the most critical limitations\nof MRI: the high number of false positives leading to an excessive number of unnecessary biopsies. These biopsies, often\nperformed under MRI guidance when feasible, significantly increase healthcare costs, prolong diagnostic timelines, and\nutilize vital radiological resources that could otherwise be allocated to more pressing clinical needs. In this context, the\ndevelopment of AI systems capable of refining diagnostic specificity, particularly for BI-RADS 4 lesions, is both relevant and\ncommendable.\nThe manuscript introduces the BI-RADS 4 Lesions Analysis System (BL4AS), an AI-based tool leveraging multiphase DCE-\nMRI and a pretrained foundation model to segment and classify breast lesions. The system demonstrates high diagnostic\naccuracy and promising potential to improve radiologists' performance in both retrospective and prospective cohorts.\nWhile the topic and approach are noteworthy, several critical concerns necessitate a thorough revision:\nAbstract\n-The abstract effectively summarizes the study's aim, methods, and key findings\n-you provided a good general introduction to the topic and well explained the results’ implications.\nMethodological Rigor:\n- Overall, the \"Methods\" section is detailed and well-structured but needs some refinement in terms of clarity, accessibility,\nand correction of typographical errors.\n-The manuscript does not provide specific details about the MRI machines used, including vendors, models, or coils, which\nare important given the variability in image quality across platforms. It is unclear whether subgroup analyses were performed\nto examine performance differences between different scanners, or if such differences were considered negligible.\n-Protocol standardization across centers is not sufficiently addressed. There is no discussion of harmonization of dynamic\nphase timing, contrast agent type and dosage, or consistency in patient preparation and acquisition sequences.\n-Greater detail is needed regarding preprocessing steps across different centers\n-the washout period may be insufficient to eliminate recall bias, especially given the small prospective dataset. Consider\ndiscussing this limitation, or justifying the chosen interval based on existing standards or prior literature.\n-Analysis is limited to the largest lesion in multifocal cases. Consider methods to analyze multiple lesions per patient in\nfuture developments to better reflect multifocal cases and clinical practice.\n- It would be beneficial to have a clear distinction between primary and secondary outcomes in the analysis and results\npresentation.\nComparison to Established Literature:\n-The paper would benefit from a more rigorous and transparent comparison with other models.\n-You should clarify why the use of a foundation model substantively improves over traditional pretrained CNNs or radiomics-\nbased methods. Additionally, the term 'foundation model' should be clearly defined within the manuscript, especially for\nreaders unfamiliar with the concept. This clarification would enhance reader understanding.\nDiscussion\n- The whole section is quite well structured however you should better elaborate on the limitations\n-The manuscript addresses a highly relevant clinical issue. Nevertheless, its utility is currently confined to BI-RADS 4 cases.\nBroader implications, such as integration into comprehensive AI-aided diagnostic pathways, should be more thoroughly\nexplored in the discussion section.\nThe manuscript could further elaborate on how such systems might support clinical practice in resource-limited settings,\nwhere variability in radiologist expertise could affect diagnostic consistency.\n-I would suggest elaborating, in this section, on the recommendations for future research\nSupport for Conclusions and Claims:\n-While the results are encouraging and indicate substantial potential, certain assertions regarding clinical readiness and\ngeneralizability may be premature. These should be more cautiously framed in light of the limited scope of the prospective\nvalidation and the need for broader multi-center real-world testing.\nBest Regards."}, "author_response": "Dear Editor and Reviewers,\nWe sincerely appreciate your constructive comments on our manuscript entitled “An Interpretable\nAI System Reduces False-positive MRI Diagnoses by Stratifying High-risk Breast Lesions”\n(Manuscript Number: NCOMMS-25-20370A).\nThe insightful suggestions have significantly helped strengthen our work. Below, we present a point-\nby-point response to each comment, elaborating on the specific revisions made. The reviewers’\ncomments are reproduced in black, our responses are in blue, and the corresponding modifications in\nthe manuscript are marked in red. We have also uploaded a revised version of the manuscript with\nall changes highlighted in yellow for each reference.\nREVIEWER COMMENTS\nReviewer #1 (Remarks to the Author):\nR1.1: You have provided an interesting study on an AI algorithm for the classification for lesions in\nbreast MRI. My major issue is that I do not really understand why you limited yourselves to bi-rads\n4 lesions. It makes much more sense to classify all lesions in the breast and aim to prevent biopsy in\na large proportion of these cases. You in essence show that that is possible, but I cannot understand\nin how many cases biopsy could simply be avoided. The classification into a/b/c is useful, but in\npractice does not lead to different approaches.\nResponse to R1.1: We appreciate the reviewer’s thoughtful comment regarding our focus on BI-\nRADS 4 lesions. The main reason is that BI-RADS 4 lesions represent the greatest diagnostic\nchallenge in breast imaging. BI-RADS 5 lesions have >95% malignancy probability, and BI-RADS\n1-3 lesions have ≤2% malignancy rate1 — both relatively straightforward to diagnose. In contrast,\nBI-RADS 4 lesions exhibit significant overlap in imaging characteristics between benign and\nmalignant lesions, accounting for the majority of false-positive biopsies (MRI-guided biopsy yields\nonly 20.5%-30.3% positive rates)2. Therefore, AI assistance is most critically needed for this category\nwhere diagnostic uncertainty is highest. Although BI-RADS 1-3 and BI-RADS 5 lesions are generally\nmore straightforward to diagnose in routine practice, their inclusion was not the primary focus of this\nstudy, as our work aimed to address the most pressing diagnostic gap. We plan to expand our analysis\nto include these categories in future research to provide a more comprehensive assessment.\nRegarding the clinical utility of A/B/C subcategorization, while current BI-RADS guidelines\nrecommend biopsy for all category 4 lesions, emerging evidence suggests more nuanced management\nstrategies may be appropriate. Flowers et al.3 demonstrated that experienced radiologists could\nprovide highly accurate risk assessment, showing that low-risk 4A lesions could be safely managed\nwith clinical assessment and follow-up rather than immediate biopsy. This approach safely reduced\nfalse-positive biopsies by 22%-56% while missing only 0%-2% of invasive cancers. Furthermore, the\nlatest BI-RADS 6th edition now formally recommends A/B/C subcategorization for MRI BI-RADS\n4 lesions. Our BL4AS system may reduce the burden on radiologists in this assessment process while\nproviding more accurate risk evaluation and reducing inter-reader variability, as demonstrated in our\nmanuscript.\nRegarding your specific question about biopsy avoidance, we analyzed different operational points\nof BL4AS based on training set to evaluate potential management strategies. Our BL4AS system\ndemonstrates significant clinical impact by avoiding 123 unnecessary biopsies per 1,000 patients\nwhile maintaining 100% sensitivity in the external test set at a high NPV operating point.\nWhile including all BI-RADS categories represents a reasonable future direction, our study aimed\nto address the diagnostic challenge where AI assistance is most critically needed. Additionally, we\nhave added this concern to our revised manuscript’s limitations section, where we discuss how\nBL4AS could be integrated into a comprehensive AI-assisted diagnostic workflow that encompasses\nall breast lesions.\nModifications in Discussion:\n“Second, our study focused exclusively on BI-RADS 4 lesions. Future research should include all BI-\nRADS categories and integrate BL4AS into a two-stage diagnostic pathway: initial breast MRI\ncomputer-aided detection for lesion identification across all categories, followed by BL4AS as a\nspecialized refinement tool for diagnostically uncertain lesions, enabling precise risk stratification\nwhile preserving workflow efficiency.”\n1. Istomin, A. et al. A multiparametric classification system for lesions detected by breast magnetic\nresonance imaging. Eur. J. Radiol. 132, 109322 (2020).\n2. Mahoney, M. C., Gatsonis, C., Hanna, L., DeMartini, W. B. & Lehman, C. Positive Predictive\nValue of BI-RADS MR Imaging. Radiology 264, 51–58 (2012).\n3. Flowers, C. I. et al. Reducing false-positive biopsies: a pilot study to reduce benign biopsy rates\nfor BI-RADS 4A/B assessments through testing risk stratification and new thresholds for\nintervention. Breast Cancer Res Tr 139, 769–777 (2013).\nR1.2: In addition, your algorithm also does segmentation, but I did not get clear what ground truth\nwas used for that, or how you envision using that. I.e. that seems to be a bit of a loose end.\nResponse to R1.2: We appreciate the reviewer’s comment regarding the segmentation component of\nour algorithm. We apologize that the annotation process was not described well in the original\nmanuscript. We have described the annotations in Supplementary Methods in the revised manuscript,\nas follows.\nModifications in Supplementary Methods:\n“Before training the segmentation model, all lesions in the training set were manually annotated by\na radiologist (R9) with over 8 years of breast MRI experience using ITK-SNAP software. To assess\ninter-observer consistency, 100 patients were randomly chosen from the training set. Another\nexperienced radiologist (R10) independently re-segmented these cases following the same annotation\ncriteria (referencing both radiological and pathological diagnostic information, including tumor\nlocation and size). The Dice similarity coefficient was used to quantify agreement between R9 and\nR10. A Dice score of 0.881 was achieved, indicating strong inter-observer segmentation consistency.\nFor intra-observer reliability, radiologist R9 repeated the segmentation of the same 100 randomly\nselected cases after a 1-month interval (to minimize recall bias). Using the Dice similarity coefficient\nagain to measure agreement with the initial annotations, a Dice score of 0.912 was obtained. This\nhigh value confirms robust intra - observer segmentation reproducibility.\nThe trained segmentation model was subsequently applied to segment lesions in other data sets. To\nminimize potential localization errors, we implemented a quality control step wherein the\nexperienced radiologist manually reviewed the segmentation results. In instances where the model\nfailed to accurately segment the tumor, the radiologist manually corrected. The quality assured center\nensures that the correct volume of interest (VOI) was cropped for downstream prediction.”\nR1.3: In general, your results section is hard to understand without the methods section. It would be\nbetter to at least introduce a little the applied methods too, because of the manuscript design of nature\ncomm.\nResponse to R1.3: We appreciate the reviewer’s constructive comment regarding the clarity of our\nresults section. We agree that the format of Nature Communications necessitates results to be more\nself-contained and accessible to readers who may not immediately consult the methods section. To\naddress this, we have revised the results section by incorporating brief methodological context—\nsufficient to enhance comprehension while avoiding duplication of the detailed procedures outlined\nin the methods section.\nR1.4: Your test-set is relatively small. That is not an issue, but when you move towards subset\nanalysis, I expect that the confidence intervals will be huge. that is not visible from Fig. 3. Please\ncheck.\nResponse to R1.4: We appreciate the reviewer’s insightful comment. We note that Fig. 3 presents\nROC curves without confidence intervals, and we suspect the reference may be to Table.3, which\ndisplays performance metrics alongside their corresponding confidence intervals. Your observation\nregarding wider confidence intervals in subset analyses is well-recognized. As evident in Table 3,\nparticularly in the stratified analysis of junior and senior readers, the confidence intervals do broaden\nsubstantially in these subset analyses. To clarify, our confidence intervals were computed using a\nbootstrap resampling method with 1,000 iterations. This approach is specifically chosen to enhance\nthe stability and reliability of estimates, even when dealing with smaller sample sizes. While sample\nsize does influence interval width, the 1,000-iteration bootstrap procedure we employed helps\nmitigate this effect, providing robust statistical inference. We have thoroughly verified the\ncalculations of all confidence intervals and confirmed their accuracy.\nTable 3 | The diagnostic performance of BL4AS, readers alone, and BL4AS-assisted readers\nin the prospective test set.\nAUC Accuracy Sensitivity Specificity PPV NPV\n(95%CI) (95%CI) (95%CI) (95%CI) (95%CI) (95%CI)\n0.892 0.870 0.862 0.889 0.949 0.727\nBL4AS\n(0.813-0.951) (0.793-0.935) (0.773-0.944) (0.759-1.000) (0.885-1.000) (0.562-0.882)\nReaders without BL4AS assistance\n0.682 0.761 0.873 0.491 0.808 0.618\nALL\n(0.622-0.829) (0.698-0.871) (0.782-0.939) (0.409-0.779) (0.745-0.921) (0.454-0.839)\n0.635 0.745 0.900 0.370 0.775 0.607\nJunior\n(0.535-0.731) (0.652-0.832) (0.828-0.974) (0.182-0.553) (0.675-0.862) (0.352-0.838)\n0.729 0.777 0.846 0.611 0.841 0.630\nSenior\n(0.634-0.835) (0.700-0.867) (0.777-0.937) (0.426-0.784) (0.751-0.921) (0.468-0.834)\nReaders with BL4AS assistance\n0.837 0.867 0.910 0.764 0.903 0.783\nALL\n(0.758-0.926) (0.807-0.939) (0.846-0.977) (0.609-0.928) (0.833-0.971) (0.645-0.945)\n0.837 0.870 0.915 0.759 0.902 0.790\nJunior\n(0.736-0.919) (0.799-0.929) (0.856-0.981) (0.564-0.891) (0.816-0.964) (0.638-0.947)\n0.836 0.864 0.904 0.769 0.904 0.775\nSenior\n(0.755-0.929) (0.802-0.935) (0.845-0.975) (0.608-0.925) (0.833-0.972) (0.646-0.938)\nA few specific comments:\nR1.5: The training and validation sets comprised 1,881 patients with 582 benign lesions, => why\ndidn’t you use malignant lesions in the training dataset?\nResponse to R1.5: We appreciate the reviewer’s comment. We apologize for the unclear expression\nin our original statement. The training and validation sets included both benign and malignant lesions.\n“1,881 patients with 582 benign lesions”, we intended to convey that among the total lesions from\n1,881 patients, 582 were benign lesions, with the remaining lesions being malignant. We recognize\nthis phrasing was ambiguous and could be misinterpreted as suggesting only benign lesions were\nincluded.\nModifications in Results:\n“The training and validation sets comprised 1,881 patients, including 582 benign lesions and 1,355\nmalignant lesions, while external tests A and B included 103 patients (61 benign lesions and 47\nmalignant lesions) and 614 patients (513 benign lesions and 153 malignant lesions), respectively.”\nR1.6: BL4AS demonstrated robust performance in lesion segmentation, achieving Dice similarity\ncoefficients of 0.851, 0.834, 0.825 and 0.837 in the training => you did not even state that the source\nimages were annotated, so how do you produce dice data. Can you elaborate on the annotation process?\nResponse to R1.6: We appreciate the reviewer’s comment. We sincerely apologize that the\nannotation process was not described well in the original manuscript. We have described the\nannotations in Supplement Methods in the revised manuscript, as follows.\nModifications in Supplement Methods:\n“Before training the segmentation model, all lesions in the training set were manually annotated by\na radiologist (R9) with over 8 years of breast MRI experience using ITK-SNAP software. To assess\ninter-observer consistency, 100 patients were randomly chosen from the training set. Another\nexperienced radiologist (R10) independently re-segmented these cases following the same annotation\ncriteria (referencing both radiological and pathological diagnostic information, including tumor\nlocation and size). The Dice similarity coefficient was used to quantify agreement between R9 and\nR10. A Dice score of 0.881 was achieved, indicating strong inter-observer segmentation consistency.\nFor intra-observer reliability, radiologist R9 repeated the segmentation of the same 100 randomly\nselected cases after a 1-month interval (to minimize recall bias). Using the Dice similarity coefficient\nagain to measure agreement with the initial annotations, a Dice score of 0.912 was obtained. This\nhigh value confirms robust intra - observer segmentation reproducibility.\nThe trained segmentation model was subsequently applied to segment lesions in other data sets. To\nminimize potential localization errors, we implemented a quality control step wherein the\nexperienced radiologist manually reviewed the segmentation results. In instances where the model\nfailed to accurately segment the tumor, the radiologist manually corrected. The quality assured center\nensures that the correct volume of interest (VOI) was cropped for downstream prediction.”\nR1.7: To evaluate real-world applicability, BL4AS was tested on a prospective test set from Center\n1, achieving an AUC of 0.892 = > you seem to test only in patients with known lesions. Did you\nactually point out the lesions to be assessed or do you do detection too?\nResponse to 1.7: We appreciate the reviewer’s comment. To clarify, our study focused on the\ndiagnostic challenge rather than lesion detection. In our prospective validation, we evaluated BL4AS\nperformance on lesions that had already been identified and categorized as BI-RADS 4 during\nstandard clinical MRI interpretation by radiologists.\nThe clinical scenario we aimed to address involves cases where radiologists have successfully\nidentified enhancing lesions but face diagnostic uncertainty in determining their malignancy risk.\nThese lesions represent cases with diagnostic uncertainty where the primary clinical challenge lies in\naccurate risk stratification rather than initial detection, as the lesions are typically readily visible on\ncontrast-enhanced MRI sequences.\nBL4AS is designed to serve as a diagnostic refinement tool for radiologists who have already\nidentified lesions but require assistance in making informed biopsy decisions, which represents a\ncommon real-world clinical scenario. As noted in our limitations, we envision BL4AS being\nintegrated into a two-stage diagnostic pathway: initial breast MRI computer-aided detection for lesion\nidentification across all BI-RADS categories, followed by BL4AS as a specialized tool for\ndiagnostically uncertain lesions to enable precise risk stratification while preserving workflow\nefficiency.\nThis focused approach allows us to validate BL4AS diagnostic capabilities in the specific clinical\ncontext where AI assistance would be most valuable - helping radiologists resolve diagnostic\nuncertainty for identified lesions rather than replacing the lesion detection process.\nR1.8: the same eight readers reviewed the same cases with BL4AS 114 assistance +> can you explain\nthe design of the reader study? Is it sequential or fully crossed. Were readers informed of the AI\nquality?\nResponse to R1.8: We appreciate the reviewer’s comment. Our study employed a fully crossed\ndesign where the same eight readers (four senior and four junior radiologists) evaluated identical\ncases under two conditions: first without AI assistance, and subsequently with BL4AS assistance after\na 4-week washout period to minimize recall bias. This design allows for direct comparison of each\nreader’s performance with and without AI support while controlling for inter-reader variability.\nBesides, all readers were blinded to the AI model’s performance in order to reduce expectation bias\nand ensure an unbiased assessment.\nModifications in Methods:\n“To assess the utility of BL4AS in aiding radiologists’ clinical decision-making, an AI-assisted reader\nstudy with a fully crossed design was conducted. The same 92 breast lesions in the prospective test\nset were presented again to the same eight readers 4 weeks after the end of the first assessment. For\neach lesion, together with the original MRI images, the corresponding AI prediction on malignancy\nrisk probability was provided to the readers at the same time. Besides, all readers were blinded to\nthe AI model’s performance in order to reduce expectation bias and ensure an unbiased assessment.”\nReviewer #2 (Remarks to the Author):\nR2.1: The manuscript presents BL4AS (BI-RADS 4 Lesions Analysis System), a deep learning-based\nAI model designed to stratify BI-RADS 4 breast lesions using dynamic contrast-enhanced MRI\n(DCE-MRI). Trained on a substantial dataset of 17,149 MRI volumes, BL4AS demonstrates high\ndiagnostic performance across multiple external and prospective test sets, achieving AUCs ranging\nfrom 0.892 to 0.930. Notably, the system significantly reduces false-positive rates and inter-reader\nvariability and enhances radiologists’ specificity and efficiency when used as a decision-support tool.\nA prospective test with 88 cases demonstrated improved diagnostic consistency among radiologists\nwhen aided by the system. The model also provides interpretable Grad-CAM visualizations to support\nclinical decision-making. Despite encouraging results, the study’s limitations include a small\nprospective sample and lack of formal adherence to reporting standards like CLAIM 2 (Checklist for\nArtificial Intelligence in Medical Imaging) guidelines [Radiology, doi: 10.1148/ryai.240300].\nResponse to R2.1: We sincerely appreciate the reviewer’s comprehensive evaluation and positive\ncomments on our work. We are pleased to see recognition of our achievements in reducing false-\npositive rates, improving inter-reader consistency, and enhancing radiologists’ diagnostic\nperformance.\nFirstly, we acknowledge that the prospective test set has a relatively limited sample size. However,\nwe would like to emphasize several important points: First, our validation strategy employed a multi-\ntiered approach, including multiple independent external test sets and prospective test set, providing\ncomprehensive assessment of model generalizability. Second, we employed a fully crossed, paired\nreader study design in the prospective test set, which allows for within-subject comparison and helps\nmaximize statistical power. Based on our sample size estimation, the prospective test set of 88 cases\nis sufficient to achieve 80% statistical power for detecting clinically meaningful differences. We fully\nagree that larger-scale prospective multicenter studies are needed to further validate BL4AS clinical\nvalue, and we have clarified this in the discussion section as future research plans.\nFurthermore, we greatly appreciate the reviewer highlighting the importance of CLAIM 2 guideline\ncompliance. In our revised manuscript, we have reviewed carefully our reporting according to the\nCLAIM 2 checklist and made corresponding improvements to enhance transparency, reproducibility,\nand clarity.\nWe believe that addressing these important issues will enable our research to make a more rigorous\nand valuable contribution to the field of AI-assisted diagnosis for BI-RADS 4 lesions. We again thank\nthe reviewer for their professional review and constructive suggestions.\nSignificance and Comparison with Established Literature\nR2.2: This work addresses a relevant clinical problem in breast imaging: the overdiagnosis and low\nspecificity of MRI in BI-RADS 4 lesions. While similar efforts have been reported (e.g., radiomics\nor CNN-based classifiers for breast MRI), few have employed foundation models with prospective\ntesting. However, key works such as Daimiel Naranjo et al. (Cancers, 2022, doi:\n10.3390/cancers14071743) and Sun et al. (J. Transl. Med., 2021, doi: 10.1186/s12967-021-03117-5)\nhave tackled similar goals.\nResponse to R2.2: We appreciate the reviewer’s acknowledgment of the clinical relevance of our\nwork. We have carefully reviewed the studies by Daimiel Naranjo et al. and Sun et al., which indeed\nprovide valuable insights into AI applications for breast MRI. However, there are several differences\nbetween our work and previous studies, which highlight our unique contributions.\nBoth referenced studies were retrospective analyses with relatively small sample sizes (fewer than\n600 lesions), whereas our study analyzed a substantially larger multicenter dataset of 2,803 lesions\nand included prospective validation with a fully crossed reader study design. More importantly, unlike\nprevious works that encompassed all BI-RADS categories, our study specifically targets BI-RADS 4\nlesions, a clinically challenging subgroup where diagnostic uncertainty is highest and accounting for\nthe majority of false-positive biopsies. By concentrating exclusively on BI-RADS 4 lesions, we could\naddress the critical unmet clinical need of reducing unnecessary biopsies in this diagnostically\nchallenging population. Additionally, our system employs a foundation model framework and\nincorporates interpretable Grad-CAM visualizations to enhance clinical decision-making, with\nvalidation conducted across multiple external test sets and prospective test set to ensure robust\ngeneralizability. We have revised our manuscript to include appropriate discussion of these relevant\nprior works, including studies by Daimiel Naranjo et al. and Sun et al., while discussing the key\ndifferences between our study and their studies.\nModifications in Discussion:\n“Previous AI-based studies, including works by Daimiel Naranjo et al.25 and Sun et al.24, have\nexplored CNN-based classifiers and radiomics approaches for breast MRI diagnosis but were limited\nby small sample sizes (<600 lesions), retrospective designs, and inclusion of all BI-RADS categories\nrather than focusing on the most challenging BI-RADS 4 subgroup.”\nSupport for Conclusions and Need for Additional Evidence\nR2.3: The conclusions are generally supported by the results, particularly in terms of improved\nspecificity and inter-reader agreement. However, the small prospective sample size (n=88)\nundermines the strength of the claims related to clinical impact. Furthermore, there is no sample size\nor power justification in the statistical methods, which is essential for validating the adequacy of the\ntest cohort.\nResponse to R2.3: We thank the reviewer for this important comment regarding sample size\njustification. We acknowledge that this information was not clearly presented in our original\nsubmission. In the revised manuscript, we have provided a detailed explanation of the sample size\nestimation.\nWe fully agree that our findings require validation in larger cohorts before drawing broader\nconclusions about clinical impact. We have added this limitation in the revised manuscript and\nemphasized that larger multicenter prospective studies will be essential to further validate the\ngeneralizability and clinical impact of these findings across diverse clinical settings and patient\npopulations.\nModifications in Supplement Methods:\n“Sample Size Calculation: For the retrospective study evaluation, sample size estimation was\nperformed using the “pROC” package in R software (version 4.5.1). Based on the following\nparameters: an expected area under the curve (AUC) value of 0.80, an alpha level of 0.05, a power\nof 0.90, a kappa value of 1.00 (determined according to the malignant lesion rate of approximately\n50%-70%1-3), and a 20% dropout rate, the minimum required sample size was calculated to be 43\ncases. In the present study, we enrolled 1,505, 376, 103, and 614 patients in the training, validation,\nand two external test sets, respectively, which exceeded the statistical requirements.\nFor the prospective multi-reader study, we performed sample size calculation using the\n“MRMCsamplesize” package in R software (version 4.5.1), which is specifically designed for multi-\nreader multi-case studies. The calculation was based on parameters derived from existing literature4\nand clinical expectations: (1) anticipated effect size of 0.1 for AUC improvement, (2) r1 = 0.6 for the\ncrossed-paired design, (3) baseline AUC of 0.8, (4) rangeb = 0.15 and rangew = 0.05, (5) R = 1, (6)\nnumber of readers = 8, (7) an alpha level of 0.05, (8) a power of 0.80, and (0) a 20% dropout rate.\nBased on these parameters, the minimum required sample size was calculated to be 80 cases. Our\nprospective test set included 88 patients, which met the statistical requirements.”\nModifications in Discussion:\n“Third, our prospective test set was relatively small and single-institutional. Future validation\nrequires larger, multicenter studies across varying medical resource levels to establish broader\nclinical applicability.”\n1. Yu, T. et al. Integrating intratumoral and peritumoral radiomics with deep transfer learning for\nDCE-MRI breast lesion differentiation: A multicenter study comparing performance with\nradiologists. European Journal of Radiology 177, 111556 (2024).\n2. Hu, Q. et al. Improved Classification of Benign and Malignant Breast Lesions Using Deep Feature\nMaximum Intensity Projection MRI in Breast Cancer Diagnosis Using Dynamic Contrast-\nenhanced MRI. Radiology: Artificial Intelligence 3, e200159 (2021).\n3. Lyu, Y. et al. Combination of ultrafast dynamic contrast-enhanced MRI-based radiomics and\nartificial neural network in assessing BI-RADS 4 breast lesions: Potential to avoid unnecessary\nbiopsies. Front. Oncol. 13, 1074060 (2023).\n4. Obuchowski, N. A. & Hillis, S. L. Sample Size Tables for Computer-Aided Detection Studies.\nAmerican Journal of Roentgenology 197, W821–W828 (2011).\nR2.4: A major omission is the lack of a detailed analysis of false negatives, which are critical in\noncologic diagnostics.\nResponse to R2.4: We thank the reviewer for this important comment. We agree that false negative\nanalysis is indeed critical in oncologic diagnostics, and we have provided an analysis of\nmisclassification cases in manuscript.\nAs described in Methods, we investigated controversial cases and specifically focused on BL4AS\nprediction errors to better understand the system’s performance limitations. This analysis was\ndesigned to identify patterns and characteristics associated with misclassification. Given the critical\nimportance of false negative cases in oncologic diagnosis, we conducted a comprehensive analysis to\nidentify patterns and characteristics associated with these misclassifications.\nOur detailed investigation revealed some imaging and pathological characteristics of false negative\ncases. From an imaging perspective, these cases predominantly exhibited cystic-solid lesions with\nmixed components, small lesions with smooth margins, and non-mass enhancement patterns that\nlacked typical malignant features. Pathologically, the misclassified cases were primarily ductal\ncarcinoma in situ (DCIS), mucinous carcinoma, and early-stage invasive carcinomas, which\ncharacteristically present with less aggressive imaging appearances and closely mimic benign lesions.\nImportantly, our analysis indicated that incorporating additional imaging sequences, particularly DWI,\ncould provide complementary diagnostic information to assist radiologists in better characterizing\nthese morphologically ambiguous cases.\nIn the revised manuscript, we provided detailed case presentations with representative imaging\nexamples, including DWI sequences and imaging feature descriptions, along with corresponding\npathological findings. This analysis offers valuable clinical insights for radiologists when interpreting\nBL4AS results in practice, particularly for lesions exhibiting these challenging imaging\ncharacteristics.\nModifications in Methods:\n“Given the critical importance of false negative cases in oncologic diagnosis, we conducted a\ncomprehensive analysis to explore the imaging and pathological features of these misclassifications.”\nModifications in Results:\n“Our detailed analysis of false negative cases revealed that these misclassifications predominantly\ninvolved cystic-solid lesions with mixed components, small lesions with smooth margins, and non-\nmass enhancement patterns lacking typical malignant features. Pathologically, BL4AS misclassified\nductal carcinoma in situ and mucinous carcinoma cases more frequently, likely due to their less\ninvasive imaging characteristics that closely mimic benign lesions. Notably, incorporating additional\nsequences particularly DWI showed potential to assist in better characterizing these morphologically\nambiguous cases (Supplementary Fig. 5-8).”\nSupplementary Fig. 5 | Representative false negative case 1: Ductal carcinoma in situ\nmisclassified by BL4AS. A 45-year-old woman with ductal carcinoma in situ presenting as a non-\nmass lesion with segmental distribution, uneven margins, heterogeneous internal enhancement and\na persistent signal enhancement time curve. The ADC value of this lesion equals to 1.728×10−3\nmm2/s.\nSupplementary Fig. 6 | Representative false negative case 2: Solid papillary carcinoma in situ\nmisclassified by BL4AS. A 58-year-old woman with solid papillary carcinoma in situ presenting as\na non-mass lesion with linear distribution, uneven margins, heterogeneous internal enhancement and\na persistent signal enhancement time curve. The ADC value of this lesion equals to 1.205×10−3 mm2/s.\nSupplementary Fig. 7 | Representative false negative case 3: Early invasive ductal carcinoma\nmisclassified by BL4AS. A 59-year-old woman with T1N0 invasive ductal carcinoma presenting as\na mass lesion with round shape, circumscribed margins, homogeneous internal enhancement and a\nwashout signal enhancement time curve. The ADC value of this lesion equals to 0.675×10−3 mm2/s.\nSupplementary Fig. 8 | Representative false negative case 4: Mucinous carcinoma misclassified by\nBL4AS. A 73-year-old woman with T2N0 mucinous carcinoma presenting as a mass lesion with\nround shape, uneven margins, heterogeneous internal enhancement and a persistent signal\nenhancement time curve. The ADC value of this lesion equals to 2.407×10−3 mm2/s.\nFlaws in Data Analysis, Interpretation, and Conclusions\nR2.5: The limited prospective cohort size and single-institution origin introduce concerns of\noverfitting and limit generalizability.\nResponse to R2.5: We thank the reviewer for this important comment. Although our prospective\ncohort size was limited and of single-institution origin, several aspects of our research design helped\nto reduce the risk of overfitting. As mentioned in our response to Response 2.3, we conducted sample\nsize calculations to ensure adequate statistical power, with both our retrospective validation and\nprospective reader studies designed to meet statistical requirements with at least 80% power. In\naddition, following the TRIPOD guidelines1, our study design corresponds to Type 3 prediction\nmodel research, which involves development of a prediction model using one data set and evaluation\nof its performance on separate data. This represents one of the most rigorous forms of model\nvalidation. Furthermore, our research utilized two independent external test sets to comprehensively\nevaluate the performance of BL4AS. This validation approach using completely separate datasets\nprovides robust evidence for model generalizability, as recommended by TRIPOD guidelines for\nhigh-quality prediction model studies. Importantly, our prospective cohort serves a critical purpose\nbeyond model validation: it evaluates the model’s real-world clinical utility and its potential to assist\nphysicians in diagnostic decision-making. Moreover, while our prospective cohort originates from a\nsingle institution, it includes data from different scanner types and field strengths (both 1.5T and 3T\nMRI scanners across different machines), which helps address technical generalizability concerns.\nBL4AS exhibited stable performance across these external test sets and the prospective test set,\nachieving AUCs ranging from 0.892 to 0.930. Overall, these results demonstrate the robustness and\ngeneralizability of BL4AS.\n1. Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. Transparent reporting of a multivariable\nprediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Bmc\nMed 13, 1 (2015).\nR2.6: Absence of formal sample size calculations and interaction testing in subgroup analysis\nweakens the methodological rigor.\nResponse to R2.6: We thank the reviewer for this important comment. Regarding the sample size\ncalculations, as mentioned in our response to Response 2.3, we have conducted formal sample size\ncalculations in the revised manuscript to ensure adequate statistical power. For the subgroup analyses,\nwe acknowledge the importance of interaction testing to assess whether the impact of BL4AS\nassistance on radiologists’ diagnostic performance differs significantly across different subgroups. In\nour revised manuscript, we have included appropriate interaction tests to statistically validate whether\nthe AI assistance effect varies between subgroups, rather than relying solely on descriptive\ncomparisons.\nModifications in Methods:\n“Sample size calculations are detailed in the Supplemental Methods. Interaction effects between AI\nassistance and subgroup variables were assessed using generalized estimating equation models.”\nModifications in Results:\n“We further evaluated the differential effects of AI assistance across various clinical scenarios using\ngeneralized estimating equation models to assess interaction effects between AI assistance and\nsubgroup variables. As shown in Supplementary Fig. 2, the analysis revealed interactions for\nsensitivity regarding age group (P = 0.015), and for specificity regarding lesion size (P = 0.010),\nenhancement pattern (P < 0.001), magnetic field strength (P = 0.002), and scanner type (P = 0.026),\nindicating that the magnitude of AI assistance benefits varied across different patient characteristics\nand imaging parameters. Despite these differential effects, BL4AS consistently improved reader\nperformance across most evaluated subgroups, including varying background parenchymal\nenhancement (BPE), age, lesion size, magnetic field strength, scanner type, and lesion enhancement\npattern (Fig. 4h, Supplementary Fig. 3).”\nSupplementary Fig. 2 | Interaction effects between AI assistance and subgroup variables in the\nprospective test set. Comparison of diagnostic performance between with (purple) or without (bule)\nBL4AS-assisted diagnosis across the different age, degree of background parenchymal enhancement,\nlesion size, lesion enhancement pattern, magnetic field strength and scanner type subgroups. P-\nvalues represent interaction effects from generalized estimating equation models.\nR2.7: Grad-CAM is a reasonable starting point but provides only limited insight into decision\nboundaries.\nResponse to R2.7: We thank the reviewer for this important comment. In our manuscript, we\nemployed Grad-CAM as a tool to enhance the interpretability of our model’s decisions, primarily by\nillustrating which areas of the images the model focuses on. However, the intrinsic limitations of\nGrad-CAM, notably its coarse spatial resolution stemming, preclude detailed pixel-level analysis.\nDespite these limitations, we selected Grad-CAM for several practical reasons: it provides clear visual\nexplanations that radiologists can easily understand and has been successfully used in many medical\nimaging studies. Our results demonstrate that Grad-CAM visualizations provided valuable insights\nin our reader study, helping radiologists understand the AI reasoning process and assess whether the\nmodel focused on appropriate diagnostic regions, thereby enhancing confidence in AI-assisted\ndecision-making.\nWe consider Grad-CAM as an initial step toward better understanding AI decision-making. Future\nstudies will investigate more advanced interpretability approaches, including attention mechanisms\nand feature attribution methods, to provide deeper insights into how models make diagnostic\ndecisions.\nR2.8: Another major limitation of the manuscript is the lack of direct comparison with existing AI\nmodels. While BL4AS demonstrates strong performance within its own dataset, the absence of a\nquantitative comparison against previously published models limits the ability to assess its true\ninnovation and clinical added value.\nResponse to R2.8: We appreciate the reviewer’s comment regarding the comparison with existing\nAI models. However, to our knowledge, there were no publicly available AI models specifically\ndesigned for breast BI-RADS 4 lesion malignancy prediction. To address this limitation and provide\nmeaningful benchmarks, we implemented and trained several commonly used methods from the\nliterature on our dataset for comparison with BL4AS.\nSpecifically, we implemented comparisons with traditional machine learning approaches (Decision\nTree, Logistic Regression, Random Forest, Support Vector Classifier, and XGBoost) using\nhandcrafted radiomics features, alternative deep learning architectures (ResNet50, DenseNet121,\nEfficientNetB0, SEResNet101, and Vision Transformer). All comparison models were trained and\nvalidated using the same dataset, standardized preprocessing pipelines and experimental protocol to\nensure fair evaluation. The results demonstrate that BL4AS achieves superior performance compared\nto these established baseline methods. The detailed comparative analysis methods and corresponding\nresults have been included in the revised manuscript.\nModifications in Methods:\n“While previous studies have proposed various AI models for breast lesion classification, none\nspecifically address BI-RADS 4 lesion malignancy prediction. To provide comprehensive\nperformance benchmarks, we implemented multiple established approaches as comparison baselines,\nincluding traditional machine learning methods using handcrafted radiomics features, and state-of-\nthe-art deep learning architectures. All baseline models were trained and evaluated using identical\ndatasets, preprocessing pipelines, and experimental protocols to ensure fair comparison. Detailed\ntraining parameters, hyperparameter settings, and implementation details for all comparison\nmethods are provided at GitHub (https://github.com/zhenweishi/BL4AS).\nModifications in Results:\n“To establish the superiority of our approach, we compared BL4AS against multiple baseline\nmethods adapted for breast lesion diagnosis. These included traditional machine learning algorithms\nutilizing handcrafted radiomics features, as well as state-of-the-art deep learning architectures. Our\nresults demonstrated that BL4AS outperformed competing methods across the majority of evaluation\nmetrics (Supplementary Table 2), with particularly notable improvements in specificity for BI-RADS\n4 lesions.”\nSupplementary Table 2 | Comparison with other methods in the retrospective dataset.\nSensitivit Specificit\nMethod AUC Accuracy PPV NPV\ny y\nTraining set\nRadiomics-Decision Tree 0.813 0.774 0.891 0.523 0.800 0.692\nRadiomics-Logistic\n0.784 0.735 0.757 0.686 0.838 0.570\nRegression\nRadiomics-Random Forest 0.852 0.772 0.757 0.805 0.892 0.608\nRadiomics-SVC 0.860 0.797 0.948 0.475 0.794 0.811\nRadiomics-XGBoost 0.906 0.814 0.800 0.844 0.916 0.664\nDenseNet121 0.879 0.813 0.835 0.765 0.884 0.684\nEfficientNetB0 0.681 0.633 0.617 0.667 0.798 0.449\nResNet50 0.811 0.736 0.726 0.757 0.865 0.564\nSEResNet101 0.798 0.774 0.849 0.611 0.824 0.655\nViT 0.742 0.625 0.534 0.819 0.863 0.451\nBL4AS (ours) 0.964 0.894 0.886 0.913 0.956 0.789\nValidation set\nRadiomics-Decision Tree 0.732 0.748 0.886 0.452 0.777 0.648\nRadiomics-Logistic\n0.778 0.725 0.727 0.722 0.849 0.552\nRegression\nRadiomics-Random Forest 0.791 0.698 0.679 0.738 0.848 0.517\nRadiomics-SVC 0.803 0.766 0.930 0.413 0.773 0.732\nRadiomics-XGBoost 0.819 0.738 0.723 0.770 0.871 0.564\nDenseNet121 0.824 0.753 0.801 0.651 0.831 0.603\nEfficientNetB0 0.673 0.660 0.672 0.635 0.798 0.473\nResNet50 0.797 0.723 0.708 0.754 0.861 0.546\nSEResNet101 0.778 0.776 0.867 0.579 0.816 0.670\nViT 0.685 0.572 0.476 0.778 0.822 0.408\nBL4AS (ours) 0.873 0.829 0.867 0.746 0.880 0.723\nExternal test set A\nRadiomics-Decision Tree 0.694 0.685 0.681 0.689 0.627 0.737\nRadiomics-Logistic\n0.607 0.565 0.298 0.770 0.500 0.588\nRegression\nRadiomics-Random Forest 0.745 0.639 0.383 0.836 0.643 0.638\nRadiomics-SVC 0.712 0.685 0.766 0.623 0.610 0.776\nRadiomics-XGBoost 0.784 0.676 0.426 0.869 0.714 0.663\nDenseNet121 0.868 0.759 0.681 0.820 0.744 0.769\nEfficientNetB0 0.671 0.611 0.489 0.705 0.561 0.642\nResNet50 0.784 0.685 0.468 0.852 0.710 0.675\nSEResNet101 0.826 0.759 0.638 0.852 0.769 0.754\nViT 0.663 0.630 0.362 0.836 0.630 0.630\nBL4AS (ours) 0.896 0.787 0.617 0.918 0.853 0.757\nExternal test set B\nRadiomics-Decision Tree 0.756 0.655 0.769 0.622 0.366 0.905\nRadiomics-Logistic\n0.727 0.748 0.578 0.796 0.445 0.869\nRegression\nRadiomics-Random Forest 0.791 0.797 0.524 0.875 0.542 0.866\nRadiomics-SVC 0.770 0.682 0.823 0.642 0.394 0.928\nRadiomics-XGBoost 0.780 0.802 0.537 0.877 0.552 0.870\nDenseNet121 0.861 0.812 0.585 0.877 0.573 0.882\nEfficientNetB0 0.667 0.655 0.442 0.715 0.305 0.819\nResNet50 0.794 0.809 0.483 0.902 0.582 0.860\nSEResNet101 0.830 0.791 0.673 0.825 0.521 0.899\nViT 0.714 0.755 0.313 0.881 0.426 0.819\nBL4AS (ours) 0.930 0.889 0.626 0.963 0.829 0.901\nR2.9: In the methods section, two critical design choices are reported without sufficient justification\nand should be clarified by the authors. First, the decision to select only the largest lesion in cases of\nmultifocal disease may introduce bias, particularly if smaller lesions have different imaging\ncharacteristics or histopathological profiles. The rationale for excluding additional lesions should be\nexplained, along with its potential impact on model generalizability and performance. Second, the\ndivision of patients from Center 1 into training and internal test sets using an 8:2 ratio, and the\nassignment of patients from Centers 2 and 3 to external test sets A and B, respectively, is described\nwithout justification. The authors should clarify whether stratification or balancing was applied\nduring the split and explain why this center-specific allocation was chosen, especially given the\npossible influence of institutional imaging protocols and patient populations on model performance.\nAdditionally, these omissions result in the lack of compliance with CLAIM items 19–21, which\nrequire a detailed explanation of data partitioning, the rationale for dataset construction, and potential\nsources of bias or confounding.\nResponse to R2.9: We appreciate the reviewer’s insightful comments regarding these critical\nmethodological considerations, which have prompted us to clarify our design choices in greater detail.\nRegarding the selection of the largest lesion in multifocal disease cases, we acknowledge that\nincorporating multiple lesions could provide additional clinical value. However, our approach was\nguided by several practical and clinical factors: First, multifocal cases represented only a small subset\nof our overall cohort, and analyzing all lesions would have significantly increased computational\ncomplexity and annotation time without proportional gains in model performance for the majority of\ncases. Second, this strategy aligns with standard clinical practice—radiologists typically prioritize the\nlargest or most suspicious lesion for biopsy and management decisions, as these are the most\nclinically impactful. Indeed, histopathological confirmation, which serves as the reference standard\nfor our model, is nearly always based on the largest lesion in multifocal presentations. This approach\nis also consistent with established practices in breast imaging AI research, where focus on the most\nsignificant lesion for model development1,2,3.\nWe recognize that excluding smaller lesions represents a limitation, particularly in fully capturing\nthe complexity of multifocal disease. To address this, we have added a discussion in the limitations\nsection and explicitly state our plan to develop methods for incorporating multiple lesions in future\nwork, which will involve advanced annotation strategies and multi-lesion modeling approaches to\nbetter reflect the full spectrum of clinical presentations.\nConcerning the data partitioning strategy, we have revised the manuscript to provide\ncomprehensive justification aligned with CLAIM guidelines items 19-21: Center 1 was chosen as the\nprimary training site due to its substantially larger sample size and diverse imaging data, which\nincluded acquisitions from 1.5T and 3.0T scanners across two manufacturers. This diversity provided\noptimal conditions for model learning. The 8:2 split within Center 1 was performed using stratified\nsampling to maintain the original benign-to-malignant ratio across training and internal test sets,\nminimizing selection bias. Centers 2 and 3 were designated as independent external test sets to\nrigorously evaluate model generalizability across different institutions with varying imaging\nprotocols and patient populations. This center-specific allocation strategy enables comprehensive\nassessment of model performance across diverse clinical environments while avoiding potential data\nleakage between training and testing phases.\nModifications in Discussion:\n“Additionally, our analysis focused on the largest lesion in multifocal cases, simplifying clinical\nreality. Future studies should develop algorithms processing multiple lesions simultaneously to better\nreflect clinical practice.”\nModifications in Methods:\n“Patients from Center 1 were randomly divided into training (80%) and internal validation (20%)\nsets using stratified sampling to maintain balanced representation of malignant and benign cases\nacross both subsets, while patients from Center 2 and Center 3 formed independent external test sets\nA and B to assess cross-institutional generalizability.”\n1. Qian, X. et al. Prospective assessment of breast cancer risk from multimodal multiview ultrasound\nimages via clinically applicable deep learning. Nat. Biomed. Eng. 5, 522–532 (2021).\n2. Qian, X. et al. A multimodal machine learning model for the stratification of breast cancer risk.\nNat. Biomed. Eng 9, 356–370 (2024).\n3. Gao, Y. et al. An explainable longitudinal multi-modal fusion model for predicting neoadjuvant\ntherapy response in women with breast cancer. Nat Commun 15, 9613 (2024).\nR2.10: The manuscript does not report whether any analysis was performed to assess differences in\nimaging data derived from 1.5T versus 3T scanners. Given that image quality, signal-to-noise ratio,\nand enhancement characteristics can vary significantly between field strengths, this may influence\nboth feature extraction and model performance. Could the authors clarify whether such a comparison\nwas conducted, and if not, provide justification for its omission?\nResponse to R2.10: We sincerely appreciate the reviewer for this insightful comment. We agree that\nvariations in image characteristics between 1.5T and 3T scanners, such as signal-to-noise ratio and\nenhancement patterns, are an important consideration that could potentially influence model\nperformance.\nIn our initial submission, we conducted a preliminary subgroup analysis on the prospective test set\nto investigate this potential effect, with the results presented in Supplementary Fig. 3. To further and\nmore comprehensively address this point, and inspired by your valuable suggestion, we have now\nexpanded this analysis. In the revised manuscript, we further performed a subgroup analysis stratified\nby scanner field strength (1.5T vs. 3T) to evaluate the model’s performance and generalizability under\nthese different imaging conditions. Corresponding results were added in the revised manuscript.\nModifications in Methods:\n“To evaluate the robustness of our findings across different clinical scenarios and technical\nparameters, we conducted subgroup analyses in the combined validation and external test sets,\nincluding age (≤50 years and >50 years), lesion size (≤2 cm and >2 cm), lesion enhancement patterns\n(mass and non-mass), BPE levels (minimal or mild and moderate or marked), magnetic field strength\n(1.5T and 3T) and scanner types (GE, Philips and SIEMENS).”\nModifications in Results:\n“To evaluate model robustness and generalizability across diverse clinical scenarios, we performed\nsubgroup analyses based on patient demographics, lesion characteristics, and imaging protocols.\nThe findings indicated that BL4AS exhibited satisfactory predictive performance across combined\nvalidation and external test sets when evaluated across different patient age groups, various lesion\ncharacteristics including size, enhancement patterns, and background parenchymal enhancement\nlevels, and diverse imaging protocols (1.5T/3T, different scanner types), as shown in Supplementary\nFig. 1.”\nSupplementary Fig. 1 | Malignancy diagnosis performance of BL4AS on key subgroups on the\ncombined validation and external test sets. AUC, accuracy, sensitivity and specificity are reported\nfor each subgroup, from left to right. The dark pink and light pink bars on the right represent the\nnumber of malignant and benign cases for each subgroup. All metrics are presented with 95% Cl\nbased on 1000-time bootstrap. AUC area under the receiver operating characteristic curve.\nData and Methodology\nR2.11: The prospective validation cohort includes only 88 cases, which is, in my opinion, insufficient\nto draw strong or generalizable clinical conclusions. This limited sample size significantly reduces\nthe statistical power of the study and raises concerns about potential selection bias. Moreover, the\nmanuscript does not include any formal justification or power analysis to validate the adequacy of\nthe sample size for prospective evaluation. Furthermore, the prospective cohort used for real-world\ntesting (Table 1) includes a disproportionate prevalence of malignant cases 65/88 (70.7%), which\ndoes not reflect the true distribution typically encountered in clinical practice for BI-RADS 4 lesions.\nThis artificially high prevalence may bias performance metrics, especially positive predictive value,\nand limits the generalizability of the findings.\nResponse to R2.11: We thank the reviewer for this important comment. We appreciate the\nopportunity to provide further clarification on the study design and the context of our results.\nAs we also detailed in our response to Response 2.3, we have performed a formal sample size\ncalculation in the revised manuscript to justify the adequacy of our cohorts. Both our retrospective\nvalidation studies and the prospective reader study were designed to achieve a statistical power of at\nleast 80%, ensuring that the sample size was sufficient to detect a significant effect and draw\nmeaningful conclusions.\nConcerning the high prevalence of malignancy in the prospective test set, we agree this is an\nimportant observation that needs for discussion. Although this rate is at a high level, it is still within\nthe risk of malignancy as defined in the ACR BI RADS guidelines for BI-RADS 4 lesions. In fact,\nseveral published studies of BI-RADS 4 lesions have reported similarly high rates of malignancy,\nusually in the range of 60 to 70 percent1,2.\nFurthermore, to address the critical issue of generalizability and minimize potential biases from\nany single cohort, our study was designed to include two additional independent external validation\ndatasets. These datasets were all established through consecutive collection, providing more\nrepresentative samples of real-world clinical scenarios at each center.\nWe acknowledge the importance of conducting large-scale, multi-center prospective studies to\nfurther validate our findings. We have added this limitation in our revised manuscript and plan to\nexpand our research in future work to include larger, more diverse patient populations across multiple\ninstitutions, which will provide more robust evidence for the clinical utility and generalizability of\nour model in real-world practice.\nModifications in Discussion:\n“Third, our prospective dataset was relatively small and single-institutional. Future validation\nrequires larger, multicenter studies across varying medical resource levels to establish broader\nclinical applicability.”\n1. Yu, T. et al. Integrating intratumoral and peritumoral radiomics with deep transfer learning for\nDCE-MRI breast lesion differentiation: A multicenter study comparing performance with\nradiologists. European Journal of Radiology 177, 111556 (2024).\n2. Hu, Q. et al. Improved Classification of Benign and Malignant Breast Lesions Using Deep Feature\nMaximum Intensity Projection MRI in Breast Cancer Diagnosis Using Dynamic Contrast-\nenhanced MRI. Radiology: Artificial Intelligence 3, e200159 (2021).\nR2.12: The manuscript does not provide a comprehensive analysis of false negative cases, which is\na critical omission. Given the high clinical risk associated with missed malignancies in breast imaging,\ndetailed examination of these errors is essential. Understanding where and why the model fails would\nbe necessary to assess its safety and clinical reliability.\nResponse to R2.12: We thank the reviewer for this important comment. We fully agree that\ncomprehensive analysis of false negative cases is essential for assessing the safety and clinical\nreliability of our model, particularly given the high clinical risk associated with missed malignancies.\nAs we detailed in our response to Response 2.4, we conducted a comprehensive analysis to identify\npatterns and characteristics associated with these misclassifications. Our detailed investigation\nrevealed some imaging and pathological characteristics of false negative cases. From an imaging\nperspective, these cases predominantly exhibited cystic-solid lesions with mixed components, small\nlesions with smooth margins, and non-mass enhancement patterns that lacked typical malignant\nfeatures. Pathologically, the misclassified cases were primarily ductal carcinoma in situ (DCIS),\nmucinous carcinoma, and early-stage invasive carcinomas, which characteristically present with less\naggressive imaging appearances and closely mimic benign lesions. Importantly, our analysis\nindicated that incorporating additional imaging sequences, particularly DWI, could provide\ncomplementary diagnostic information to assist radiologists in better characterizing these\nmorphologically ambiguous cases.\nIn the revised manuscript, we provided detailed case presentations with representative imaging\nexamples, including DWI sequences and radiological feature descriptions, along with corresponding\npathological findings. This analysis offers valuable clinical insights for radiologists when interpreting\nBL4AS results in practice, particularly for lesions exhibiting these challenging imaging\ncharacteristics.\nModifications in Methods:\n“Given the critical importance of false negative cases in oncologic diagnosis, we conducted a\ncomprehensive analysis to explore the imaging and pathological features of these misclassifications.”\nModifications in Results:\n“Our detailed analysis of false negative cases revealed that these misclassifications predominantly\ninvolved cystic-solid lesions with mixed components, small lesions with smooth margins, and non-\nmass enhancement patterns lacking typical malignant features. Pathologically, BL4AS misclassified\nductal carcinoma in situ and mucinous carcinoma cases more frequently, likely due to their less\ninvasive imaging characteristics that closely mimic benign lesions. Notably, incorporating additional\nsequences particularly DWI showed potential to assist in better characterizing these morphologically\nambiguous cases (Supplementary Fig. 5-8).”\nSupplementary Fig. 5 | Representative false negative case 1: Ductal carcinoma in situ\nmisclassified by BL4AS. A 45-year-old woman with ductal carcinoma in situ presenting as a non-\nmass lesion with segmental distribution, uneven margins, heterogeneous internal enhancement and\na persistent signal enhancement time curve. The ADC value of this lesion equals to 1.728×10−3\nmm2/s.\nSupplementary Fig. 6 | Representative false negative case 2: Solid papillary carcinoma in situ\nmisclassified by BL4AS. A 58-year-old woman with solid papillary carcinoma in situ presenting as\na non-mass lesion with linear distribution, uneven margins, heterogeneous internal enhancement and\na persistent signal enhancement time curve. The ADC value of this lesion equals to 1.205×10−3 mm2/s.\nSupplementary Fig. 7 | Representative false negative case 3: Early invasive ductal carcinoma\nmisclassified by BL4AS. A 59-year-old woman with T1N0 invasive ductal carcinoma presenting as\na mass lesion with round shape, circumscribed margins, homogeneous internal enhancement and a\nwashout signal enhancement time curve. The ADC value of this lesion equals to 0.675×10−3 mm2/s.\nSupplementary Fig. 8 | Representative false negative case 4: Mucinous carcinoma misclassified by\nBL4AS. A 73-year-old woman with T2N0 mucinous carcinoma presenting as a mass lesion with\nround shape, uneven margins, heterogeneous internal enhancement and a persistent signal\nenhancement time curve. The ADC value of this lesion equals to 2.407×10−3 mm2/s.\nAnalytical Approach\nR2.13: The analytical approach is sound, employing appropriate performance metrics (AUC,\nspecificity, sensitivity) and comparative analyses with radiologists’ assessments.\nResponse to R2.13: We thank the reviewer for this positive comment regarding our analytical\napproach and the appropriateness of our performance metrics and comparative analyses.\nR2.14: Discuss the rationale behind chosen thresholds for classification and their clinical implications.\nResponse to R2.14: We thank the reviewer for this constructive comment. Our threshold selection\nstrategy was designed to balance statistical rigor with clinical utility, providing flexible decision-\nmaking tools for diverse clinical scenarios. The primary classification threshold was determined using\nthe maximum Youden index from the training set, which optimizes the balance between sensitivity\nand specificity for overall diagnostic performance. Subsequently, recognizing the complexity of BI-\nRADS 4 lesion management, we developed different operating points to accommodate different\nclinical needs. As described in our methods, we evaluated two operational points: one optimized for\nhigh positive predictive value (PPV) to identify high-risk lesions requiring immediate intervention,\nand another optimized for high negative predictive value (NPV) to potentially identify low-risk cases\nsuitable for active surveillance. The high-NPV threshold (maintaining 100% sensitivity) was selected\nto address the critical clinical need to avoid missing malignancies while potentially reducing\nunnecessary biopsies. At this operating point, BL4AS could prevent 123 unnecessary biopsies per\n1,000 patients in the external test set while maintaining diagnostic safety. Conversely, the high-PPV\nthreshold was designed to ensure prompt identification and intervention for high-risk lesions.\nModifications in Methods:\n“To evaluate potential management strategies, we analyzed different operational points of BL4AS.\nIn particular, we focused on two scenarios. First, the high-NPV threshold was designed to maintain\n100% sensitivity (no missed malignancies) while maximizing the identification of truly benign lesions\nthat could potentially avoid immediate biopsy. This approach addresses the clinical need to reduce\npatient anxiety and healthcare costs associated with unnecessary procedures while ensuring\ndiagnostic safety. Second, the high-PPV threshold was optimized to minimize false positive\npredictions, thereby ensuring that lesions flagged as high-risk truly require immediate intervention.\nThis dual-threshold strategy accommodates different clinical scenarios and risk-benefit\nconsiderations in management decisions.”\nR2.15: Consider performing subgroup analyses based on lesion characteristics or patient\ndemographics to explore model performance across different scenarios.\nResponse to R2.15: We thank the reviewer for this constructive comment. We fully agree that\nsubgroup analyses allow readers to more directly understand the potential practice scope and\nboundaries of the model. In our initial submission, we conducted subgroup analyses based on key\nlesion characteristics and patient demographics on the prospective test set, with results presented in\nFig.4 and Supplementary Fig. 2.\nFollowing your recommendation, we have now extended these subgroup analyses in our study. We\nreported the model performances comprehensively stratified by patient age, lesion characteristics\n(including size, enhancement patterns, and background parenchymal enhancement levels).\nCorresponding results were added in Supplementary Fig. 1.\nModifications in Methods:\n“To evaluate the robustness of our findings across different clinical scenarios and technical\nparameters, we conducted subgroup analyses in the combined validation and external test sets,\nincluding age (≤50 years and >50 years), lesion size (≤2 cm and >2 cm), lesion enhancement patterns\n(mass and non-mass), BPE levels (minimal or mild and moderate or marked), magnetic field strength\n(1.5T and 3T) and scanner types (GE, Philips and SIEMENS).”\nModifications in Results:\n“To evaluate model robustness and generalizability across diverse clinical scenarios, we performed\nsubgroup analyses based on patient demographics, lesion characteristics, and imaging protocols.\nThe findings indicated that BL4AS exhibited satisfactory predictive performance across combined\nvalidation and external test sets when evaluated across different patient age groups, various lesion\ncharacteristics including size, enhancement patterns, and background parenchymal enhancement\nlevels, and diverse imaging protocols (1.5T/3T, different scanner types), as shown in Supplementary\nFig. 1.”\nSupplementary Fig. 1 | Malignancy diagnosis performance of BL4AS on key subgroups on the\ncombined validation and external test sets. AUC, accuracy, sensitivity and specificity are reported\nfor each subgroup, from left to right. The dark pink and light pink bars on the right represent the\nnumber of malignant and benign cases for each subgroup. All metrics are presented with 95% Cl\nbased on 1000-time bootstrap. AUC area under the receiver operating characteristic curve.\nReviewer #2 (Remarks on code availability):\nR2.16: Upon reviewing the BL4AS GitHub repository (https://github.com/zhenweishi/BL4AS), to\nenhance the reproducibility and utility of the BL4AS codebase for the research community, I\nrecommend the following improvements:\n- Expand the README to include detailed installation instructions, usage examples, and explanations\nof the code structure.\n- Provide a requirements.txt or environment.yml file to facilitate the setup of the necessary Python\nenvironment.\n- Include information on how to access the datasets used or provide scripts for data preprocessing and\nformatting.\n- Offer example scripts or notebooks demonstrating how to train the model, evaluate its performance,\nand interpret the results.\nResponse to R2.16: We sincerely appreciate the reviewer’s thorough examination of our BL4AS\nGitHub repository and the constructive suggestions for improving code reproducibility and\naccessibility for the research community. We acknowledge that the current repository could benefit\nsignificantly from enhanced documentation and user-friendly features to facilitate adoption and\nvalidation by other researchers. We are committed to implementing all the recommended\nimprovements and will update the repository accordingly. Specifically, we will expand the README\nfile to include comprehensive installation instructions with step-by-step setup guides, detailed usage\nexamples with sample code snippets, clear explanations of the code architecture and module functions,\nand troubleshooting guides for common issues. We will provide both requirements.txt and\nenvironment.yml files to ensure seamless environment setup across different systems, along with\ndependency version specifications to guarantee compatibility. Regarding data accessibility, while\npatient privacy regulations prevent direct sharing of clinical datasets, we will provide detailed data\npreprocessing scripts, data format specifications and requirements, synthetic data examples for testing\npurposes, and clear documentation of the data structure and annotation protocols used in our study.\nAdditionally, we will create comprehensive example notebooks demonstrating the complete\nworkflow from data loading through model training, evaluation metrics calculation, result\nvisualization, and clinical interpretation of outputs. We will also include pre-trained model weights\nand inference scripts to enable immediate testing and validation by the research community.\nReviewer #3 (Remarks to the Author):\nR3.1: The study presents an innovative and timely contribution to breast imaging by addressing one\nof the most critical limitations of MRI: the high number of false positives leading to an excessive\nnumber of unnecessary biopsies. These biopsies, often performed under MRI guidance when feasible,\nsignificantly increase healthcare costs, prolong diagnostic timelines, and utilize vital radiological\nresources that could otherwise be allocated to more pressing clinical needs. In this context, the\ndevelopment of AI systems capable of refining diagnostic specificity, particularly for BI-RADS 4\nlesions, is both relevant and commendable.\nThe manuscript introduces the BI-RADS 4 Lesions Analysis System (BL4AS), an AI-based tool\nleveraging multiphase DCE-MRI and a pretrained foundation model to segment and classify breast\nlesions. The system demonstrates high diagnostic accuracy and promising potential to improve\nradiologists' performance in both retrospective and prospective cohorts. While the topic and approach\nare noteworthy, several critical concerns necessitate a thorough revision.\nResponse to R3.1: Thank you very much for your thoughtful and encouraging comments. We are\npleased that you recognize the clinical significance of our work in addressing the critical challenge\nof high false-positive rates in breast MRI, particularly for BI-RADS 4 lesions. We appreciate your\nrecognition of the BL4AS system’s performance and its potential to enhance radiologists’ diagnostic\ncapabilities in clinical practice. We have carefully replied to your comments point-by-point as follows.\nAbstract\nR3.2: The abstract effectively summarizes the study's aim, methods, and key findings; You provided\na good general introduction to the topic and well explained the results’ implications.\nResponse to R3.2: We thank the reviewer for this positive comment on our abstract and introduction,\nand our efforts to clearly communicate the study’s objectives and clinical implications.\nMethodological Rigor:\nR3.3: Overall, the \"Methods\" section is detailed and well-structured but needs some refinement in\nterms of clarity, accessibility, and correction of typographical errors.\nResponse to R3.3: We thank the reviewer for this positive comment. We have revised this section to\nimprove its logical flow and coherence, employing clearer figures and descriptions to enhance\nreadability. The modifications are highlighted in the revised manuscript.\nR3.4: The manuscript does not provide specific details about the MRI machines used, including\nvendors, models, or coils, which are important given the variability in image quality across platforms.\nIt is unclear whether subgroup analyses were performed to examine performance differences between\ndifferent scanners, or if such differences were considered negligible.\nResponse to R3.4: We thank the reviewer for this important comment. In the revised manuscript, we\nhave provided comprehensive details about the MRI machines used across all data sets, including\nspecific information about vendors, models, and coil configurations.\nIn our initial submission, we conducted a preliminary subgroup analysis on the prospective test set\nto examine performance differences between different scanners, with the results presented in\nSupplementary Fig. 2. To further and more comprehensively address this point, and inspired by your\nvaluable suggestion, we have expanded this analysis. In the revised manuscript, we performed a\nthorough subgroup analysis stratified by different scanners to evaluate the model’s performance and\ngeneralizability under these different imaging conditions. Corresponding results were added in the\nrevised manuscript.\nModifications in Supplement Methods:\n“Contrast-enhanced T1-weighted imaging (T1 +C) images with fat suppression were used in this\nstudy, which were extracted using 1.5T or 3.0T scanners with a dedicated bilateral breast coil. A\ngadolinium-based agent (Magnevist, Bayer Healthcare, Berlin, Germany) was injected through an\nMRI compatible power injector at a rate of 2 mL/s and at a dose of 0.2 mL/kg of body weight, followed\nby a 20-mL saline flush with high-pressure injector. The acquisition parameters of the three centers\nare shown in Table S3. The data acquisition procedure was described as follows:\nFor center 1, all MRI scans were obtained using 1.5T or 3.0T Philips (Achieva, Philips Healthcare,\nBest, the Netherlands; Ingenia, Philips Healthcare, Best, the Netherlands) or 1.5T GE (Optima\nMR360; GE HealthCare, Milwaukee, WI, USA) MRI scanners. An initial fat-saturated T1-weighted\nimaging (T1WI) precontrast scan was collected before contrast medium injection. The dynamic T1\n+C images included five post contrast scans, which were collected with an interval of 60 seconds\nafter the intravenous injection of gadolinium contrast agent.\nFor center 2, all MRI scans were obtained using 1.5T or 3.0T Philips (Achieva, Philips Healthcare,\nBest, the Netherlands; Ingenia, Philips Healthcare, Best, the Netherlands) or 3.0T Siemens (Siemens\nSkyra, Siemens Healthcare, Erlangen, Germany) MRI scanners. An initial fat-saturated T1-weighted\nimaging (T1WI) precontrast scan was collected before contrast medium injection. For Philips\nscanners, the dynamic T1+C images included five post-contrast scans, while for Siemens scanners,\nsix post-contrast scans were acquired, all collected with an interval of 60 seconds after the\nintravenous injection of gadolinium contrast agent.\nFor center 3, all MRI scans were obtained using 3.0T Philips (Ingenia, Philips Healthcare, Best,\nthe Netherlands) or 1.5T GE (Brivo MR360, GE HealthCare, Milwaukee, WI, USA) MRI scanners.\nAn initial fat-saturated T1-weighted imaging (T1WI) precontrast scan was collected before contrast\nmedium injection. For Philips scanners, the dynamic T1+C images included five post-contrast scans,\nwhile for GE scanners, six post-contrast scans were acquired, all collected with an interval of 60\nseconds after the intravenous injection of gadolinium contrast agent.”\nModifications in Methods:\n“To evaluate the robustness of our findings across different clinical scenarios and technical\nparameters, we conducted subgroup analyses in the combined validation and external test sets,\nincluding age (≤50 years and >50 years), lesion size (≤2 cm and >2 cm), lesion enhancement patterns\n(mass and non-mass), BPE levels (minimal or mild and moderate or marked), magnetic field strength\n(1.5T and 3T) and scanner types (GE, Philips and SIEMENS).”\nModifications in Results:\n“To evaluate model robustness and generalizability across diverse clinical scenarios, we performed\nsubgroup analyses based on patient demographics, lesion characteristics, and imaging protocols.\nThe findings indicated that BL4AS exhibited satisfactory predictive performance across combined\nvalidation and external test sets when evaluated across different patient age groups, various lesion\ncharacteristics including size, enhancement patterns, and background parenchymal enhancement\nlevels, and diverse imaging protocols (1.5T/3T, different scanner types), as shown in Supplementary\nFig. 1.”\nSupplementary Fig. 1 | Malignancy diagnosis performance of BL4AS on key subgroups on the\ncombined validation and external test sets. AUC, accuracy, sensitivity and specificity are reported\nfor each subgroup, from left to right. The dark pink and light pink bars on the right represent the\nnumber of malignant and benign cases for each subgroup. All metrics are presented with 95% Cl\nbased on 1000-time bootstrap. AUC area under the receiver operating characteristic curve.\nR3.5: Protocol standardization across centers is not sufficiently addressed. There is no discussion of\nharmonization of dynamic phase timing, contrast agent type and dosage, or consistency in patient\npreparation and acquisition sequences.\nResponse to R3.5: We thank the reviewer for this important comment. While we acknowledge that\ncomplete protocol harmonization would be ideal from a technical perspective, we believe our\napproach of embracing protocol diversity actually represents a significant strength of our study design\nthat enhances real-world applicability.\nRegarding the key protocol parameters mentioned by the reviewer, we maintained standardization\nin standardization\nin the most critical aspects. All centers used gadolinium-based contrast agent (Magnevist, Bayer\nHealthcare, Berlin, Germany) with standardized administration parameters: injection rate of 2 mL/s\nat a dose of 0.2 mL/kg of body weight through an MRI-compatible power injector. Despite variations\nin the total number of dynamic phases across scanner types (as detailed in Supplement methods),\nwe\nstandardized the critical timing for image analysis by consistently using images acquired at 120\nseconds post-injection for peak contrast enhancement assessment and 300 seconds post-injection for\ndelayed post-contrast evaluation across all centers. Additionally, all patients underwent identical\npreparation protocols including prone positioning with dedicated breast coils.\nRather than viewing protocol variations as limitations, we leveraged the inherent diversity in\ndynamic phase timing and acquisition parameters across different centers to develop and validate a\nmore robust, generalizable AI system. As demonstrated by results across two independent test sets\nwith different institutional protocols, BL4AS maintained robust performance with AUC values of\n0.896-0.930 despite protocol variations. Our subgroup analyses across different scanner types (GE,\nPhilips, Siemens) and field strengths (1.5T vs 3.0T) showed consistent performance. This approach\nreflects the reality of multi-institutional clinical implementation, where perfect protocol\nstandardization is often impractical, and demonstrates that BL4AS can perform reliably across the\nspectrum of clinically acceptable imaging protocols encountered in routine practice.\nR3.6: Greater detail is needed regarding preprocessing steps across different centers\nResponse to R3.6: Thank you for the valuable suggestion. To ensure consistency across multicenter\ndata while accommodating protocol variations, we implemented a comprehensive standardized\npreprocessing pipeline using identical tools and parameters across all centers. This standardized\npreprocessing approach is consistent with other high-quality multicenter breast MRI AI studies2-3,\nwhich is essential for maintaining data consistency and ensuring reliable model performance across\ndifferent imaging protocols and scanner types.\nImportantly, all preprocessing parameters were determined and fixed based solely on the training\ndataset, ensuring that no information from validation or test sets influenced the preprocessing pipeline.\nThis approach eliminates any potential data leakage concerns and maintains the integrity of our\nindependent validation methodology. The specific details are described as follows.\nThank you for the valuable suggestion. To ensure consistency across multicenter data while\naccommodating protocol variations, we implemented a comprehensive standardized preprocessing\npipeline using identical software tools and parameters across all centers. The specific details are\ndescribed as follows.\nModifications in Methods:\n“Standardized preprocessing pipeline to ensure consistency across multicenter data while\naccommodating protocol variations, a comprehensive standardized preprocessing pipeline was\nimplemented using identical software tools and parameters across all centers: (1) Alignment and\nRegistration: Spatial alignment of DCE-MRI sequences using the open‑source deedsBCV package\n(https://github.com/mattiaspaul/deedsBCV) in its default setting. (3) N4 Bias Field Correction:\nIntensity inhomogeneity correction using SimpleITK’s N4BiasFieldCorrectionImageFilter in its\ndefault setting. (4) Resampling: Standardization to 1×1×1 mm³ isotropic voxel spacing. Gray‑scale\ndata employed third‑order B‑spline interpolation and binary masks were resampled with\nnearest‑neighbor interpolation to avoid label mixing. (5) Intensity Normalization: All intensity values\nwere normalized using the mean and s.d. calculated from the training set to minimize interscanner\nintensity discrepancies.”\n1. Obuchowski, N. A. & Hillis, S. L. Sample Size Tables for Computer-Aided Detection Studies.\nAmerican Journal of Roentgenology 197, W821–W828 (2011).\n2. Gao, Y. et al. An explainable longitudinal multi-modal fusion model for predicting neoadjuvant\ntherapy response in women with breast cancer. Nat Commun 15, 9613 (2024).\n3. Mao, N. et al. A multimodal and fully automated system for prediction of pathological complete\nresponse to neoadjuvant chemotherapy in breast cancer. S c i e n c e A d v a n c e s (2025).\nR3.7: the washout period may be insufficient to eliminate recall bias, especially given the small\nprospective dataset. Consider discussing this limitation, or justifying the chosen interval based on\nexisting standards or prior literature.\nResponse to R3.7: We thank the reviewer for this important comment. We would like to address this\nconcern with the following clarifications: First, to minimize potential recall bias, the order of image\nanalysis was randomized for each reading session, ensuring that readers could not predict or recall\nthe sequence from previous evaluations. Second, our chosen 4-week washout period aligns with\nestablished regulatory guidelines. according to FDA guidance, reading sessions should be separated\nby at least four weeks to avoid memory bias when the same cases are read multiple times1. While the\nguidance acknowledges that longer intervals may be advisable in some contexts, we selected the 4-\nweek minimum based on the specific characteristics of our study population and imaging task. Given\nthat our participating radiologists routinely interpret numerous breast MRI cases in their daily clinical\npractice, we determined that the 4-week interval would be sufficient to minimize recall bias in this\ncontext.\n1. Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices\nApplied to Radiology Images and Radiology Device Data in Premarket Notification (510(k))\nSubmissions. Center for Devices and Radiological Health. https://www.fda.gov/regulatory-\ninformation/searchfda-guidance-documents/clinical-performance-assessment-\nconsiderationscomputer-assisted-detection-devices-applied-radiology. Posted September 2022.\nAccessed March 2024.\nR3.8: Analysis is limited to the largest lesion in multifocal cases. Consider methods to analyze\nmultiple lesions per patient in future developments to better reflect multifocal cases and clinical\npractice.\nResponse to R3.8: We thank the reviewer for this important comment. The reason that we focused\non the largest lesion in multifocal disease cases was guided by several practical and clinical factors:\nFirst, multifocal cases represented only a small subset of our overall cohort, and analyzing all lesions\nwould have significantly increased computational complexity and annotation time without\nproportional gains in model performance for the majority of cases. Second, this strategy aligns with\nstandard clinical practice — radiologists typically prioritize the largest or most suspicious lesion for\nbiopsy and management decisions, as these are the most clinically impactful. Indeed,\nhistopathological confirmation, which serves as the reference standard for our model, is nearly always\nbased on the largest lesion in multifocal presentations. This approach is also consistent with\nestablished practices in breast imaging AI research, where focus on the most significant lesion for\nmodel development1,2,3.\nWe recognize that excluding smaller lesions represents a limitation, particularly in fully capturing\nthe complexity of multifocal disease. To address this, we have added a discussion in the limitations\nsection and explicitly state our plan to develop methods for incorporating multiple lesions in future\nwork, which will involve advanced annotation strategies and multi-lesion modeling approaches to\nbetter reflect the full spectrum of clinical presentations.\nModifications in Discussion:\n“Additionally, our analysis focused on the largest lesion in multifocal cases, simplifying clinical\nreality. Future studies should develop algorithms processing multiple lesions simultaneously to better\nreflect clinical practice.”\n1. Qian, X. et al. Prospective assessment of breast cancer risk from multimodal multiview ultrasound\nimages via clinically applicable deep learning. Nat. Biomed. Eng. 5, 522–532 (2021).\n2. Qian, X. et al. A multimodal machine learning model for the stratification of breast cancer risk.\nNat. Biomed. Eng 9, 356–370 (2024).\n3. Gao, Y. et al. An explainable longitudinal multi-modal fusion model for predicting neoadjuvant\ntherapy response in women with breast cancer. Nat Commun 15, 9613 (2024).\nR3.9: It would be beneficial to have a clear distinction between primary and secondary outcomes in\nthe analysis and results presentation.\nResponse to R3.9: Thank you for this important suggestion. We agree that clearly distinguishing\nbetween primary and secondary outcomes enhances the clarity and scientific rigor of our study. We\nwill revise our methods and results sections accordingly. Specifically, we have explicitly defined our\nprimary outcome as performance of BL4AS for differentiating malignant from benign BI-RADS 4\nbreast lesions, with secondary outcomes including (1) radiologists’ diagnostic performance\nimprovement and efficiency enhancement when assisted by BL4AS, and (2) evaluation of model’s\nclinical management strategies and utility in routine practice. In the results presentation, we will lead\nwith the primary outcome findings and then systematically present secondary outcomes in a logical\nsequence, ensuring readers can clearly understand the hierarchy of evidence supporting our\nconclusions. We believe that this restructuring will enhance the clarity of our study design and\nfacilitate more appropriate interpretation of our findings.\nModifications in Methods:\n“The performance of all models used for the classification tasks was evaluated in terms of AUC,\naccuracy, sensitivity, specificity, PPV and NPV. The primary outcome was the diagnostic\nperformance of BL4AS for classifying BI-RADS 4 breast lesions, with secondary outcomes including\n(1) improvement in radiologists’ diagnostic performance and reading efficiency when assisted by\nBL4AS and (2) assessment of the model’s potential for clinical management and routine practice\nutility.”\nComparison to Established Literature:\nR3.10: The paper would benefit from a more rigorous and transparent comparison with other models.\nResponse to R3.10: We appreciate the reviewer’s concern regarding the comparison with existing\nAI models. However, to our knowledge, there were no publicly available AI models specifically\ndesigned for breast BI-RADS 4 lesion malignancy prediction. To address this limitation and provide\nmeaningful benchmarks, we implemented and trained several commonly used methods from the\nliterature on our dataset for comparison with BL4AS.\nSpecifically, we implemented comparisons with traditional machine learning approaches (Decision\nTree, Logistic Regression, Random Forest, Support Vector Classifier, and XGBoost) using\nhandcrafted radiomics features, alternative deep learning architectures (ResNet50, DenseNet121,\nEfficientNetB0, SEResNet101, and Vision Transformer). All comparison models were trained and\nvalidated using the same dataset, standardized preprocessing pipelines and experimental protocol to\nensure fair evaluation. The results demonstrate that BL4AS achieves superior performance compared\nto these established baseline methods. The detailed comparative analysis methods and corresponding\nresults have been included in the revised manuscript.\nModifications in Methods:\n“While previous studies have proposed various AI models for breast lesion classification, none\nspecifically address BI-RADS 4 lesion malignancy prediction. To provide comprehensive\nperformance benchmarks, we implemented multiple established approaches as comparison baselines,\nincluding traditional machine learning methods using handcrafted radiomics features, and state-of-\nthe-art deep learning architectures. All baseline models were trained and evaluated using identical\ndatasets, preprocessing pipelines, and experimental protocols to ensure fair comparison. Detailed\ntraining parameters, hyperparameter settings, and implementation details for all comparison\nmethods are provided at GitHub (https://github.com/zhenweishi/BL4AS).”\nModifications in Results:\n“To establish the superiority of our approach, we compared BL4AS against multiple baseline\nmethods adapted for breast lesion diagnosis. These included traditional machine learning algorithms\nutilizing handcrafted radiomics features, as well as state-of-the-art deep learning architectures. Our\nresults demonstrated that BL4AS outperformed competing methods across the majority of evaluation\nmetrics (Supplementary Table 2), with particularly notable improvements in specificity for BI-RADS\n4 lesions.”\nSupplementary Table 2 | Comparison with other methods in the retrospective dataset.\nSensitivit Specificit\nMethod AUC Accuracy PPV NPV\ny y\nTraining set\nRadiomics-Decision Tree 0.813 0.774 0.891 0.523 0.800 0.692\nRadiomics-Logistic\n0.784 0.735 0.757 0.686 0.838 0.570\nRegression\nRadiomics-Random Forest 0.852 0.772 0.757 0.805 0.892 0.608\nRadiomics-SVC 0.860 0.797 0.948 0.475 0.794 0.811\nRadiomics-XGBoost 0.906 0.814 0.800 0.844 0.916 0.664\nDenseNet121 0.879 0.813 0.835 0.765 0.884 0.684\nEfficientNetB0 0.681 0.633 0.617 0.667 0.798 0.449\nResNet50 0.811 0.736 0.726 0.757 0.865 0.564\nSEResNet101 0.798 0.774 0.849 0.611 0.824 0.655\nViT 0.742 0.625 0.534 0.819 0.863 0.451\nBL4AS (ours) 0.964 0.894 0.886 0.913 0.956 0.789\nValidation set\nRadiomics-Decision Tree 0.732 0.748 0.886 0.452 0.777 0.648\nRadiomics-Logistic\n0.778 0.725 0.727 0.722 0.849 0.552\nRegression\nRadiomics-Random Forest 0.791 0.698 0.679 0.738 0.848 0.517\nRadiomics-SVC 0.803 0.766 0.930 0.413 0.773 0.732\nRadiomics-XGBoost 0.819 0.738 0.723 0.770 0.871 0.564\nDenseNet121 0.824 0.753 0.801 0.651 0.831 0.603\nEfficientNetB0 0.673 0.660 0.672 0.635 0.798 0.473\nResNet50 0.797 0.723 0.708 0.754 0.861 0.546\nSEResNet101 0.778 0.776 0.867 0.579 0.816 0.670\nViT 0.685 0.572 0.476 0.778 0.822 0.408\nBL4AS (ours) 0.873 0.829 0.867 0.746 0.880 0.723\nExternal test set A\nRadiomics-Decision Tree 0.694 0.685 0.681 0.689 0.627 0.737\nRadiomics-Logistic\n0.607 0.565 0.298 0.770 0.500 0.588\nRegression\nRadiomics-Random Forest 0.745 0.639 0.383 0.836 0.643 0.638\nRadiomics-SVC 0.712 0.685 0.766 0.623 0.610 0.776\nRadiomics-XGBoost 0.784 0.676 0.426 0.869 0.714 0.663\nDenseNet121 0.868 0.759 0.681 0.820 0.744 0.769\nEfficientNetB0 0.671 0.611 0.489 0.705 0.561 0.642\nResNet50 0.784 0.685 0.468 0.852 0.710 0.675\nSEResNet101 0.826 0.759 0.638 0.852 0.769 0.754\nViT 0.663 0.630 0.362 0.836 0.630 0.630\nBL4AS (ours) 0.896 0.787 0.617 0.918 0.853 0.757\nExternal test set B\nRadiomics-Decision Tree 0.756 0.655 0.769 0.622 0.366 0.905\nRadiomics-Logistic\n0.727 0.748 0.578 0.796 0.445 0.869\nRegression\nRadiomics-Random Forest 0.791 0.797 0.524 0.875 0.542 0.866\nRadiomics-SVC 0.770 0.682 0.823 0.642 0.394 0.928\nRadiomics-XGBoost 0.780 0.802 0.537 0.877 0.552 0.870\nDenseNet121 0.861 0.812 0.585 0.877 0.573 0.882\nEfficientNetB0 0.667 0.655 0.442 0.715 0.305 0.819\nResNet50 0.794 0.809 0.483 0.902 0.582 0.860\nSEResNet101 0.830 0.791 0.673 0.825 0.521 0.899\nViT 0.714 0.755 0.313 0.881 0.426 0.819\nBL4AS (ours) 0.930 0.889 0.626 0.963 0.829 0.901\nR3.11: You should clarify why the use of a foundation model substantively improves over traditional\npretrained CNNs or radiomics-based methods. Additionally, the term 'foundation model' should be\nclearly defined within the manuscript, especially for readers unfamiliar with the concept. This\nclarification would enhance reader understanding.\nResponse to R3.11: Thank you for this important suggestion. We have already addressed both\naspects mentioned in your feedback. The term \"foundation model\" is clearly defined in our Methods\nsection, where we explain that foundation models are large-scale, pre-trained models trained on\ndiverse datasets that can be adapted for various downstream tasks through fine-tuning. Additionally,\nwe have discussed the substantive improvements of foundation models over traditional pre-trained\nCNNs and radiomics-based methods in our Discussion section, highlighting their superior feature\nrepresentation and enhanced transfer learning capabilities.\nModifications in Methods:\n“BL4AS is a deep learning system designed for two primary tasks: lesion segmentation and lesion\nclassification (benign vs. malignant). Foundation models are large-scale, pre-trained models trained\non diverse datasets that can be adapted for various downstream tasks through fine-tuning. Both tasks\nwere built upon a pretrained foundation model to enhance generalizability and performance.”\nModifications in Discussion:\n“In contrast, our study comprises the largest sample size to date with 2,803 BI-RADS 4 lesions,\nincluding 774 from external multicenter test sets and 92 from a prospective test set. Unlike other\nstudies that predominantly employ radiomics or CNN classifiers, our research leverages a foundation\nmodel approach, which benefits from large-scale pre-training and enhanced feature representation\ncapabilities to better capture complex imaging patterns and improve generalizability across different\nimaging protocols and institutions. Our comparative experimental analysis demonstrates that the\nfoundation model-based approach achieves superior and more stable performance compared to\ntraditional CNN and radiomics methods across different data sets. BL4AS achieved AUCs of 0.896,\n0.930 and 0.892 in external test set A, external test set B and prospective test set, respectively,\ndemonstrating excellent diagnostic performance and robustness across multiple validation cohorts.”\nDiscussion\nR3.12: The whole section is quite well structured however you should better elaborate on the\nlimitations\nResponse to R3.12: Thank you for your thoughtful review and for recognizing the strengths of our\nDiscussion section structure. We have expanded the limitations section with more detailed discussion\nof potential constraints and future research directions as detailed in the responses below.\nR3.13: The manuscript addresses a highly relevant clinical issue. Nevertheless, its utility is currently\nconfined to BI-RADS 4 cases. Broader implications, such as integration into comprehensive AI-aided\ndiagnostic pathways, should be more thoroughly explored in the discussion section.\nResponse to R3.13: We appreciate the reviewer’s constructive comment. While our current study\nfocuses on BI-RADS 4 lesions, we have recognized the importance of addressing broader clinical\nintegration. In response to this feedback, we have expanded our discussion to explore how BL4AS\ncan be integrated into comprehensive AI-aided diagnostic pathways.\nModifications in Discussion:\n“Second, our study focused exclusively on BI-RADS 4 lesions. Future research should include all BI-\nRADS categories and integrate BL4AS into a two-stage diagnostic pathway: initial breast MRI\ncomputer-aided detection for lesion identification across all categories, followed by BL4AS as a\nspecialized refinement tool for diagnostically uncertain lesions, enabling precise risk stratification\nwhile preserving workflow efficiency.”\nR3.14: The manuscript could further elaborate on how such systems might support clinical practice\nin resource-limited settings, where variability in radiologist expertise could affect diagnostic\nconsistency.\nResponse to R3.14: We appreciate the reviewer’s valuable comment. In environments where\nradiologist expertise varies significantly, our AI-driven classification system could offer several key\nadvantages for improving diagnostic consistency. The BL4AS framework provides standardized,\nobjective assessment criteria that can help bridge expertise gaps by offering consistent risk\nstratification regardless of individual radiologist experience levels. This is particularly valuable in\nsettings where specialized breast imaging expertise may be limited, as the system can serve as a\ndecision support tool that provides consistent second opinion to less experienced radiologists.\nFurthermore, our approach could reduce inter-observer variability, which is often more pronounced\nin resource-constrained environments where training opportunities and number of cases may be\nlimited. The system’s ability to provide consistent BI-RADS 4 sub-stratifications could help\nstandardize referral patterns and treatment recommendations, ultimately improving patient outcomes\nin these challenging clinical environments. We will incorporate this important perspective into our\ndiscussion section to better highlight the potential impact of our work in addressing healthcare\ndisparities in resource-limited regions for breast cancer diagnosis.\nModifications in Discussion:\n“To ensure a fair comparison between AI and clinicians, the AI system should be implemented in a\nprospective setting5,6. Therefore, BL4AS was tested and compared with eight readers in a prospective\ntest set collected after the development of the system. In our study, BL4AS demonstrated superior\naccuracy and specificity while maintaining comparable sensitivity to senior readers. The\ninterpretation of breast MRI is often subject to significant interobserver variability7, and this\nchallenge is particularly pronounced in resource-limited settings where radiologist expertise may\nvary significantly. BL4AS could address this critical need by providing consistent second opinion that\ncould bridge expertise gaps and offer consistent risk stratification regardless of individual radiologist\nexperience levels. This is evidenced by the substantial improvement in junior readers’ performance,\nwhose accuracy improved from 0.745 to 0.870, sensitivity from 0.900 to 0.915, and specificity from\n0.370 to 0.759. This improvement was maintained even in challenging cases, such as breasts with\nmoderate or marked BPE and non-mass enhancement lesions, where interpretation is typically more\ndifficult. Furthermore, BL4AS significantly increased inter-observer agreement, with the kappa\ncoefficient rising from 0.501 to 0.746. Such improvement suggests that BL4AS has the potential to\nserve as a valuable decision support tool, particularly in environments where specialized breast\nimaging expertise may be limited, potentially offering more standardized breast MRI interpretation\nand enhance diagnostic reliability. However, further validation through multicenter studies with\nlarger sample sizes across diverse clinical settings and varying resource levels would be necessary\nto fully establish its effectiveness and generalizability across different healthcare environments.”\nR3.15: I would suggest elaborating, in this section, on the recommendations for future research\nResponse to R3.15: Thank you for this important suggestion. We expanded the limitation section to\ninclude more specific and actionable directions for future studies.\nModifications in Discussion:\n“Our study has several limitations. First, BL4AS utilized only multiphase DCE-MRI images. Future\nwork should systematically explore incorporating T2WI and DWI through multi-modal fusion\napproaches to enhance diagnostic performance. Second, our study focused exclusively on BI-RADS\n4 lesions. Future research should include all BI-RADS categories and integrate BL4AS into a two-\nstage diagnostic pathway: initial breast MRI computer-aided detection for lesion identification\nacross all categories, followed by BL4AS as a specialized refinement tool for diagnostically uncertain\nlesions, enabling precise risk stratification while preserving workflow efficiency. Third, our\nprospective dataset was relatively small and single-institutional. Future validation requires larger,\nmulticenter studies across varying medical resource levels to establish broader clinical applicability.\nAdditionally, our analysis focused on the largest lesion in multifocal cases, simplifying clinical reality.\nFuture studies should develop algorithms processing multiple lesions simultaneously to better reflect\nclinical practice. Finally, while Grad-CAM provides attention visualizations, it offers limited\nmechanistic insight. Future research should integrate multi-omics data and develop sophisticated\ninterpretability methods to establish clearer links between imaging features, molecular pathways,\nand clinical outcomes.”\nSupport for Conclusions and Claims:\nR3.16: While the results are encouraging and indicate substantial potential, certain assertions\nregarding clinical readiness and generalizability may be premature. These should be more cautiously\nframed in light of the limited scope of the prospective validation and the need for broader multi-center\nreal-world testing.\nResponse to R3.16: We appreciate the reviewer’s constructive comment. We have revised the\nmanuscript to more cautiously frame statements regarding clinical readiness, emphasizing the need\nfor broader multi-center real-world validation before clinical implementation. Additionally, we have\nexplicitly included this limitation in our discussion section, acknowledging that while our results are\nencouraging and demonstrate substantial potential, further multi-center prospective studies with\nlarger cohorts are necessary to fully validate the clinical readiness and generalizability of BL4AS\nacross diverse clinical settings and patient populations.\nModifications in Discussion:\n“Third, our prospective dataset was relatively small and single-institutional. Future validation\nrequires larger, multicenter studies across varying medical resource levels to establish broader\nclinical applicability.”\n“BL4AS represents a noninvasive, AI-driven solution for segmentation, classification, and risk\nstratification of BI-RADS 4 lesions. By leveraging multiphase DCE-MRI and explainable AI, BL4AS\nnot only improves diagnostic accuracy but also reduces unnecessary biopsies and inter-reader\nvariability. Its superior performance across multicenter and prospective validation underscores its\npotential to enhance clinical decision-making and patient outcomes. Future studies incorporating\nbroader lesion types and larger multi-center real-world testing are required to confirm its clinical\nvalidity and clinical utility.”"}, {"round": 2, "reviewer_comments": {"reviewer_1": "I have no further comments. Thank you for the elaborate explanations.", "reviewer_2": "Second round comments\nI appreciate the authors’ responses to our initial comments and the substantial improvements in the manuscript.\nNevertheless, a few technical and formal aspects still require clarification or further detail before the paper can be\nconsidered for acceptance. My main concerns are outlined below.\nWhile the authors now report inter- and intra-observer Dice coefficients, it is still unclear at which stage the manual\nannotations were performed (prior to model training, or retrospectively on model outputs). Additionally, it should be specified\nwhether the corrected model-generated masks were reintroduced into the training set or used only for validation.\nThe authors indicate that formal sample-size justifications have been added, but no numerical details are provided in the\nrebuttal or in the revised text. Please explicitly report:\n• the estimated number of cases/readers required for each experiment;\n• the actual number included;\n• whether the target statistical power was met for both retrospective and prospective settings.\nThe response on FN analysis remains qualitative and general. A more detailed breakdown would be helpful, including\nlesion size, morphology and BI-RADS 4 subcategory distribution for the false negatives.\nThere is no discussion of potential demographic or protocol-related biases. Please expand on differences in patient\npopulation and MRI protocols between the retrospective and prospective cohorts and plans for external, multi-centre\nvalidation (to support generalizability).\nSome sentences in the revised manuscript remain overly long and contain repeated concepts (e.g., threshold selection and\nclinical scenarios are explained in three separate sections). Please consider condensing and unifying these descriptions.", "reviewer_3": "Dear colleagues, you provided a thorough revision of the manuscript. You have comprehensively addressed step by step\neach comment and suggestion."}, "author_response": "Dear Editor and Reviewers,\nWe sincerely appreciate your constructive comments on our manuscript entitled “An Interpretable\nAI System Reduces False-positive MRI Diagnoses by Stratifying High-risk Breast Lesions”\n(Manuscript Number: NCOMMS-25-20370B).\nThe insightful suggestions have significantly helped strengthen our work. Below, we present a point-\nby-point response to each comment, elaborating on the specific revisions made. The reviewers’\ncomments are reproduced in black, our responses are in blue, and the corresponding modifications in\nthe manuscript are marked in red. We have also uploaded a revised version of the manuscript with\nall changes highlighted in yellow for each reference.\nREVIEWER COMMENTS\nReviewer #1 (Remarks to the Author):\nR1.1: I have no further comments. Thank you for the elaborate explanations.\nResponse to R1.1: We are extremely grateful for your review and valuable feedback on our\nmanuscript. Thank you for the constructive comments and suggestions during the review process,\nwhich have greatly helped us improve the quality of our manuscript.\nReviewer #2 (Remarks to the Author):\nR2.1: I appreciate the authors’ responses to our initial comments and the substantial improvements\nin the manuscript. Nevertheless, a few technical and formal aspects still require clarification or further\ndetail before the paper can be considered for acceptance.\nResponse to R2.1: We sincerely appreciate your continued efforts in reviewing our manuscript and\nyour recognition of the substantial improvements we have made in response to the initial comments.\nWe have carefully addressed each of your remaining technical and formal concerns in the detailed\nresponses below.\nR2.2: While the authors now report inter- and intra-observer Dice coefficients, it is still unclear at\nwhich stage the manual annotations were performed (prior to model training, or retrospectively on\nmodel outputs). Additionally, it should be specified whether the corrected model-generated masks\nwere reintroduced into the training set or used only for validation.\nResponse to R2.2: We thank the reviewer for this comment. We apologize for the confusion in our\ninitial description and provide the following detailed clarifications.\nThe manual annotations were performed prior to segmentation model training for the training dataset.\nAs described in Supplementary Methods: “Before training the segmentation model, all lesions in the\ntraining set were manually annotated by a radiologist (R9) with over 8 years of breast MRI experience\nusing ITK-SNAP software.”\nThe corrected model-generated masks were incorporated into the downstream classification analysis\nacross all datasets, including the training set. Our actual workflow follows this sequence: (1) manual\nannotation of training data by R9 → (2) segmentation model training → (3) model inference on both\ntraining and test datasets → (4) radiologist review and manual correction of all model-generated\nsegmentations (both training and validation sets) → (5) use of quality-assured segmentations for\ndownstream classification analysis.\nModifications in Methods:\n“To ensure segmentation accuracy across all datasets, an experienced radiologist reviewed and\nmanually corrected all model-generated segmentations (both training and test sets) when necessary\nbefore cropping the volumes of interest. The correct volumes of interest were subsequently used for\nfurther classification analysis.”\nModifications in Supplementary Methods:\n“After segmentation model training, the model was applied to generate predictions for all data sets\n(both training and test sets). To ensure segmentation quality, we implemented a comprehensive\nquality control step wherein the experienced radiologist manually reviewed and corrected all model-\ngenerated segmentations across all datasets. This quality assurance process ensures that accurate\nvolumes of interest (VOI) were cropped for downstream classification analysis.”\nR2.3: The authors indicate that formal sample-size justifications have been added, but no numerical\ndetails are provided in the rebuttal or in the revised text. Please explicitly report:\n• the estimated number of cases/readers required for each experiment;\n• the actual number included;\n• whether the target statistical power was met for both retrospective and prospective settings.\nResponse to R2.3: We thank the reviewer for this important comment regarding explicit numerical\ndetails of our sample size justifications. We provide the comprehensive numerical details below and\nhave enhanced the clarity of our sample size reporting in the revised manuscript.\n(1) Retrospective Study:\nRequired sample size: 43 cases (calculated using pROC package, power = 0.90, α = 0.05,\nexpected AUC = 0.80, 20% dropout rate); Actual sample size: 2,598 cases (training: 1,505;\nvalidation: 376; external test A: 103; external test B: 614), met the statistical requirements\n(2) Prospective Multi-reader Study:\nRequired sample size: 80 cases with 8 readers (calculated using MRMCsamplesize package, power\n= 0.80, α = 0.05, expected AUC improvement = 0.1, 20% dropout rate); Actual sample size: 88 cases\nwith 8 readers, met the statistical requirements\nModifications in Supplementary Methods:\n“For the retrospective study evaluation: Sample size estimation was performed using the \"pROC\"\npackage in R software (version 4.5.1). Based on the following parameters: an expected area under\nthe curve (AUC) value of 0.80, an alpha level of 0.05, a power of 0.90, a kappa value of 1.00\n(determined according to the malignant lesion rate of approximately 50%-70%1-3), and a 20%\ndropout rate, the minimum required sample size was calculated to be 43 cases. In the present study,\nwe enrolled 1,505, 376, 103, and 614 patients in the training, validation, and two external test sets,\nrespectively, which substantially exceeded the statistical requirements.\nFor the prospective multi-reader study: We performed sample size calculation using the\n\"MRMCsamplesize\" package in R software (version 4.5.1), which is specifically designed for multi-\nreader multi-case studies. The calculation was based on parameters derived from existing literature4\nand clinical expectations: (1) anticipated effect size of 0.1 for AUC improvement, (2) r1 = 0.6 for the\ncrossed-paired design, (3) baseline AUC of 0.8, (4) rangeb = 0.15 and rangew = 0.05, (5) R = 1, (6)\nnumber of readers = 8, (7) an alpha level of 0.05, (8) a power of 0.80, and (9) a 20% dropout rate.\nBased on these parameters, the minimum required sample size was calculated to be 80 cases with 8\nreaders. Our prospective test set included 88 patients and 8 readers, which met the statistical\nrequirements.”\nR2.4: The response on FN analysis remains qualitative and general. A more detailed breakdown\nwould be helpful, including lesion size, morphology and BI-RADS 4 subcategory distribution for the\nfalse negatives.\nResponse to R2.4: We thank the reviewer for this important feedback. We acknowledge that our\nprevious analysis of false negative cases was slightly qualitative and general. We have now conducted\na comprehensive quantitative analysis of false negative cases in all retrospective and prospective data\nsets, providing detailed breakdown by lesion size, morphology, and BI-RADS 4 subcategory\ndistribution as requested.\nThis quantitative analysis provides concrete insights into the specific characteristics of lesions that\nare more prone to misclassification by our model, which has important implications for clinical\nimplementation and future model improvements.\nModifications in Results:\n“However, some misclassifications occurred, particularly for lesions with peripheral enhancement\nor overlapping benign-malignant features (Supplementary Fig. 4). Our detailed analysis of false\nnegative cases revealed that these misclassifications predominantly involved small lesions,\ncircumscribed margins, and BI-RADS 4a subcategory (Supplementary Table 5).”\nSupplementary Table 5 | Detailed characteristics of false negative cases\nCharacteristics Number of cases\nDiameter (%)\n≤ 20 152 (62.8)\n> 20 90 (37.2)\nBackground parenchymal enhancement (%)\nMinimal/mild 176 (72.7)\nModerate/ marked 66 (27.3)\nType of enhancement (%)\nMass 125 (51.7)\nNon-mass 117 (48.3)\nMargin (%)\nCircumscribed 120 (49.6)\nUneven 79 (32.6)\nSpiculated 43 (17.8)\nTime-intensity curve (%)\nPersistent 64 (26.5)\nPlateau 153 (63.2)\nWashout 25 (10.3)\nBI-RADS 4 subcategory (%)\n4a 107 (44.2)\n4b 89 (36.8)\n4c 46 (19.0)\nR2.5: There is no discussion of potential demographic or protocol-related biases. Please expand on\ndifferences in patient population and MRI protocols between the retrospective and prospective\ncohorts and plans for external, multi-center validation (to support generalizability).\nResponse to R2.5: We thank the reviewer for this important comment regarding potential biases that\ncould affect generalizability. We acknowledge demographic differences between the retrospective\nand prospective datasets, systematically analyzed in Supplementary Table 1, with the prospective\ndataset demonstrating older median age. Our prospective validation was recruited from Center 1 using\nidentical imaging protocols to minimize confounding technical variables, though we recognize this\nsingle-center design may limit cross-institutional generalizability assessment. Nevertheless, our\nsubgroup analyses demonstrate robust BL4AS performance across different age groups and clinical\nscenarios (Supplementary Fig. 1), suggesting resilience to the observed demographic variations. We\nacknowledge that comprehensive multi-center prospective validation represents the definitive\napproach for establishing clinical generalizability, and we have described this limitation and outlined\nthe need for future multi-center prospective studies in our discussion section.\nModifications in Results:\n“Baseline characteristics of the patients across all datasets are summarized in Table 1, while detailed\ncomparisons between retrospective and prospective datasets are presented in Supplementary Table\n1.”\nSupplementary Table 1 | Baseline characteristics of patients between retrospective and\nprospective datasets\nRetrospective dataset Prospective dataset P value\nNumber of patients 2598 88\nAge (years, mean [IQR]) 47.00 [40.00, 56.00] 51.00 [44.75, 60.00]\n<0.001+\nMenopausal status (%)\n0.002$\nAfter menopause 881 (33.9) 45 (51.1)\nBefore menopause 1656 (63.7) 43 (48.9)\nUnknown 61 (2.4) 0 (0.0)\nNumber of lesions 2711 92\nHistological type (%)\nMalignant (%) 1555 (57.4) 65 (70.7)\n<0.001$\nInvasive ductal carcinoma 1078 (69.3) 48 (73.8)\nDuctal carcinoma in situ 305 (19.6) 5 (7.7)\nInvasive lobular carcinoma 21 (1.4) 6 (9.2)\nOthers 151 (9.7) 6 (9.2)\nBenign (%) 1156 (42.6) 27 (29.3)\n0.991$\nFibroadenoma 335 (29.0) 7 (25.9)\nAdenosis 234 (20.2) 6 (22.2)\nPapilloma 418 (36.2) 10 (37.0)\nInflammation 57 (4.9) 1 (3.7)\nOthers 64 (11.0) 3 (11.1)\nIQR interquartile range.\n+ Kruskal-Wallis test.\n$ Fisher’s exact test or Pearson’s Chi-square test.\nModifications in Discussion:\n“Third, our prospective test set was relatively small and single-institutional. The observed differences\nin baseline characteristics compared to retrospective cohorts may limit generalizability across\ndiverse clinical settings. Future validation requires larger, multicenter prospective studies across\ndiverse patient demographics and imaging protocols to establish broader clinical applicability and\nconfirm BL4AS robustness in heterogeneous real-world environments.”\nR2.6: Some sentences in the revised manuscript remain overly long and contain repeated concepts\n(e.g., threshold selection and clinical scenarios are explained in three separate sections). Please\nconsider condensing and unifying these descriptions.\nResponse to R2.6: We thank the reviewer for this valuable editorial feedback. We have carefully\nreviewed the entire manuscript to identify and address these issues. We have identified and condensed\nlengthy sentences throughout the manuscript, breaking complex compound sentences into clearer,\nmore digestible statements. Additionally, we have removed repetitive explanations of threshold\nselection and clinical scenarios that previously appeared in multiple sections (Methods, Results, and\nDiscussion) to avoid redundancy.\nReviewer #2 (Remarks on code availability):\nR2.7: The GitHub repository (https://github.com/zhenweishi/BL4AS) includes the core scripts,\npretrained weights, and an example dataset. A README file is provided with installation instructions\nand a brief description of dependencies, which are sufficient for a technically proficient user to set up\nthe environment. The provided demo successfully reproduces the core inference workflow and key\nperformance metrics reported in the manuscript, although full replication of the results would require\naccess to the original datasets. The code is structured, commented, and usable as a resource for the\ncommunity, particularly for research on BI-RADS 4 lesion stratification. Additional examples and\nextended documentation could further improve accessibility for less experienced users.\nResponse to R2.7: We thank the reviewer for the thorough evaluation of our code repository and the\npositive feedback regarding its structure, documentation, and reproducibility. We are pleased that the\nreviewer found the implementation to be well-organized and successfully able to reproduce the core\ninference workflow and key performance metrics.\nReviewer #3 (Remarks to the Author):\nR3.1: Dear colleagues, you provided a thorough revision of the manuscript. You have\ncomprehensively addressed step by step each comment and suggestion.\nResponse to R3.1: Thank you very much for your positive feedback and for acknowledging the\nimprovements in our manuscript. We are delighted to hear that our revisions have addressed your\ncomments satisfactorily. We greatly appreciate your time and effort in reviewing our manuscript and\nproviding valuable suggestions. Your insights have significantly contributed to enhancing the quality\nof our work."}]

Decision letter