Construction of a Standardized Benchmark Evaluation System for Intelligent Breast Ultrasound Image Interpretation and Systematic Performance Assessment of Multimodal Artificial Intelligence Models Based on ACR BI-RADS v2025 Criteria
Trial ID: NCT07500428
Status: RECRUITING
Study type: OBSERVATIONAL
Start date: 2026-03-12
Completion date: 2027-03-01
Sponsor: Peking Union Medical College Hospital
Collaborators: Chinese Academy of Medical Sciences
Conditions: Breast Neoplasms; Breast Diseases; Ultrasonography
Interventions: Multimodal AI Model Diagnostic Evaluation
Locations: Peking Union Medical College Hospital
Countries: China
Enrollment: 1380
Age range: 18 Years - 75 Years
Sex: FEMALE

Summary:
This single-center, retrospective, observational study aims to construct a standardized benchmark evaluation system for intelligent breast ultrasound image interpretation and to systematically assess the diagnostic performance of current mainstream multimodal artificial intelligence (AI) models.

De-identified B-mode breast ultrasound images with confirmed pathological diagnoses will be retrospectively collected from the institutional archive (2018-2025) and supplemented with images from published open-access datasets. Expert radiologists with varying experience levels will independently annotate all images according to the American College of Radiology (ACR) Breast Imaging Reporting and Data System (BI-RADS) v2025 criteria, including glandular tissue composition, lesion characterization (mass vs. non-mass lesion), morphological descriptors, and final BI-RADS classification.

Baseline deep learning models (CNN-based ResNet-50 and Transformer-based USFM) will be trained to establish performance baselines and to stratify cases by diagnostic difficulty through cross-architecture consensus. Multiple multimodal large language models (MLLMs), including both general-purpose and medical-domain models, will then be evaluated via standardized API calls using BI-RADS-guided chain-of-thought prompts at temperature 0 for reproducibility.

Primary endpoints include BI-RADS classification accuracy and diagnostic AUC for benign-malignant differentiation. Model robustness and safety will be assessed through out-of-distribution rejection testing, temperature-stability experiments, and thinking-mode ablation studies. This study adheres to the FLAIR and TRIPOD-LLM reporting guidelines.

Eligibility criteria:
Inclusion Criteria:

* B-mode breast ultrasound grayscale images from the institutional PACS database or from published open-access breast ultrasound datasets with documented original institutional ethics approval
* Image quality adequate for clinical diagnosis with clear visualization of the region of interest
* Pathological diagnosis confirmed (for benign and malignant lesion groups), or normal breast status confirmed by a senior radiologist with \>15 years of breast ultrasound experience (for the normal group)
* Complete de-identification with removal of all personally identifiable information

Exclusion Criteria:

* Severely degraded image quality precluding meaningful BI-RADS assessment
* Duplicate images from the same patient (only the most representative image retained per lesion)
* Images with residual personally identifiable information after de-identification processing
* Cases with ambiguous, disputed, or unavailable pathological results
* Non-B-mode ultrasound images, including elastography, contrast-enhanced ultrasound, and Doppler imaging

Outcomes:
Diagnostic Accuracy for Pathological Diagnosis; BI-RADS Classification Accuracy