SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subgroup Analysis. In Figures 8 and 9, we report the results of the AI model performance at detecting masses, when the model is trained and tested on the our dataset (see Section 5.1 for details of splits). We find that masses with larger sizes or higher densities (Figures 8a-b) are more easily detected. Although models trained on all sizes or mass densities have the highest performance, when the models are trained on smaller masses or lower densities, they generalize better to other masses (more difficult cases). The performance of the models are highest when they are tested and trained on the same breast density and decrease as the density of the test breast phantom differs from the train phantom (Figures 8c). The dose levels applied in this study have minimal impact on the performance
of the models and resulted in similar AUC values (Figures 8d). Evaluation of the performance change across all the breast densities (Figures 9a-b) reveals that the AUC improves with larger mass density and mass size, yet is impacted by the breast density, where mass detection performance is lowest in high-density breasts (dense) and highest in low-density breasts (fatty) in most of the cases, consistent with findings from clinical practice.
(a) AUC as a function of mass size across all breast densities.
(b) AUC as a function of mass density across all breast densities.
Figure 9: Subgroup analysis. Performance changes for models trained and tested on our M-SYNTH dataset. For each data point, the model is trained on 250 images with masses of radii of  $7\mathrm{mm}$  and mass densities of 1.06, and tested on 50 images with mass characteristics shown in plots for each specific breast density. The radiation dose level remains constant at  $100\%$  of the clinically recommended dose for each breast density during training and test.
Patient Data Evaluation. In Figure 10, we report experiments where the AI model is trained on INBreast data and evaluated on the M-SYNTH data. Although the performance results for all experiments are lower in general, we find a similar set of trends as when the model is trained on M-SYNTH data. Note that we have made no attempt to match the radiation dose levels or the image acquisition parameters for these comparisons using patient images. Even though the simulated pipeline is designed to replicate a specific DM system with a particular detector technology and technique factors, the comparison suggests similarity between the datasets. The images are qualitatively different but overall have similar glandular patterns which is an important consideration for the realism of the task of detecting masses in a noisy background. We also assessed similarity between INBreast and M-SYNTH datasets in terms of low-level pixel distributions using first five statistical moments: mean, variance, skewness, kurtosis, and hyperskewness. We found that there is a reasonably good alignment in terms of moments, especially when the synthetic images were included at all four breast densities (see supplementary material). Future work should develop a more detailed comparison including radiomics features for the training and testing datasets used in the study to complement the validation of our approach.
Limitations. There are a number of limitations to our work. First, simulations may require long runtimes and demand large computational resources, thus somewhat limiting the amounts of data that can be generated. This limitation needs to be considered with respect to the difficulty of obtaining large patient image datasets with known mass locations. In addition, data can be pre-generated offline (as we do with the M-SYNTH dataset), therefore, removing the large runtime limit and computational burden off the user. Second, testing with simulations is constrained to the variability captured by the parameter space of the object models for anatomy and pathology and the acquisition system. Thus, the complexity of the object model and acquisition system may need to be adjusted depending on the complexity of the questions to be investigated with simulated testing. In particular, a potential risk of testing using simulated data is missing the variability observed in patient populations. Finally, there is a risk of misjudging model performance due to a domain gap between real and synthetic
Figure 10: Model Evaluation. Performance changes for a model trained on 410 real patient images (INBreast dataset) and tested on our M-SYNTH dataset. The test sets consist of 50 images using parameters shown in the plots. The test radiation dose is set to  $100\%$  of the clinically recommended dose for each breast density.
examples. However, the realism and sophistication of object-based modeling of the imaging pipeline is improving rapidly and may soon compete with other approaches, making approaches based on synthetic data useful and practical for regulatory evaluation of AI-enabled medical devices.
We introduce and discuss an approach for validating AI models using physics-based simulations of digital humans from the object space to the image data, specifically for the task of breast cancer mass detection. The simulated images are highly realistic and offer a challenging test case for AI model evaluation. Our findings are consistent with expected performance and show that the AI model performance increases with mass size and mass density as expected. Finally, we show that our approach can be used to validate a model trained on independent patient data. This finding suggests that the proposed simulation setup can be used as a framework for more general evaluation of medical AI devices. The goal of this study is to demonstrate as proof-of-concept the feasibility of using simulated data to evaluate the comparative performance of AI models. In future work, it would be important to assess the evaluation approach for additional parameters in terms of the distribution of the population of digital humans in the object space, and for a range of image acquisition systems (e.g., by considering alternative simulators). By imaging a more diverse population of breast models, we hope to identify additional insights regarding AI evaluation. Finally, it is important to note that the testing is limited to the variability captured in the digital representations and may not fully indicate absolute real-world performance or trends. This study illustrates that physics-based simulation of mammography images can represent a less burdensome and cost-efficient approach for the evaluation of AI model performance across a wide range of scenarios, including a variety of image acquisition parameters and diverse populations that may not be available or are hard to obtain from human studies. Moreover, this approach offers a complementary evaluation paradigm that does not depend on the availability of patient data.
We thank Andreu Badal (OSEL/CDRH/FDA) and anonymous reviewers for helpful suggestions, Kenny Cha, Mike Mikailov and the OpenHPC team (OSEL/CDRH/FDA) for providing help with experiments, Akhonda, Mohammad (OSEL/CDRH/FDA) for help with data release, and Andreaa Kim
(OSEL/CDRH/FDA) for rendering visualizations of the 3D breast model. This is a contribution of the US Food and Drug Administration and is not subject to copyright. The mention of commercial products herein is not to be construed as either an actual or implied endorsement of such products by the Department of Health and Human Services.
[37] Brandon D. Gallas, Andriy Bandos, Frank W. Samuelson, and Robert F. Wagner. A framework for random-effects roc analysis: Biases with the bootstrap and other variance estimators. Communications in Statistics - Theory and Methods, 38(15):2586-2603, 2009.
[38] RST Catalog. iMRMC: Software for the Statistical Analysis of multi-reader multi-case studies, June 2022.
[39] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. IEEE, 2009.