Knowledge-based in silico models and dataset for the comparative evaluation of mammography AI for a range of breast characteristics, lesion conspicuities and doses
Authors: Elena Sizikova
Niloufar Saharkhiz
Diksha Sharma
Miguel Lago
Berkman Sahiner
Jana Gut Delfino
Aldo Badano
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: synthetic data; medical imaging; AI testing; digital twins
TL;DR: We release M-SYNTH, a novel dataset that relies on an in silico, physics-based simulation pipeline for comparatively evaluating the performance of AI software as a medical device for mammographic breast cancer detection.

Abstract: To generate evidence regarding the safety and efficacy of artificial intelligence (AI) enabled medical devices, AI models need to be evaluated on a diverse population of patient cases, some of which may not be readily available. We propose an evaluation approach for testing medical imaging AI models that relies on in silico imaging pipelines in which stochastic digital models of human anatomy (in object space) with and without pathology are imaged using a digital replica imaging acquisition system to generate realistic synthetic image datasets. Here, we release M-SYNTH, a dataset of cohorts with four breast fibroglandular density distributions imaged at different exposure levels using Monte Carlo x-ray simulations with the publicly available Virtual Imaging Clinical Trial for Regulatory Evaluation (VICTRE) toolkit. We utilize the synthetic dataset to analyze AI model performance and find that model performance decreases with increasing breast density and increases with higher mass density, as expected. As exposure levels decrease, AI model performance drops with the highest performance achieved at exposure levels lower than the nominal recommended dose for the breast type.

Knowledge-based in silico models and dataset for the comparative evaluation of mammography AI for a range of breast characteristics, lesion conspicuities and doses
E. Sizikova, N. Saharkhiz, D. Sharma, M. Lago, B. Sahiner, J. G. Delfino, A. Badano Office of Science and Engineering Laboratories Center for Devices and Radiological Health U.S. Food and Drug Administration Silver Spring. MD 20993 USA
Abstract
To generate evidence regarding the safety and efficacy of artificial intelligence (AI) enabled medical devices, AI models need to be evaluated on a diverse population of patient cases, some of which may not be readily available. We propose an evaluation approach for testing medical imaging AI models that relies on in silico imaging pipelines in which stochastic digital models human anatomy (in object space) with and without pathology are imaged using digital replica imaging acquisition system to generate realistic synthetic image datasets. Here, we release M-SYNTH', dataset of cohorts with four breast fibroglandular density distributions imaged at different exposure levels using Monte Carlo x-ray simulations with the publicly available Virtual Imaging Clinical Trial for Regulatory Evaluation (VICTRE) toolkit. We utilize the synthetic dataset to analyze AI model performance and find that model performance decreases with increasing breast density and increases with higher mass density, as expected. As exposure levels decrease, Al model performance drops with the highest performance achieved exposure levels lower than the nominal recommended dose for the breast type.
Mas Present7
MASS
Figure 1: Overview of the computational pipeline components for generating the M-SYNTH in silico dataset for medical imaging AI evaluation.
Introduction
The goal of this work is to demonstrate that AI models for medical imaging can be evaluated using simulations, specifically, using an in silico (also known as synthetic) imaging pipeline equipped with stochastic model for human anatomy and disease [1]. We show that in silico methods can constitute rich sources of data with realistic physical variability for performing comparative analysis of AI device performance.
"Code and data links available at: https://githaei http://sttttta.eoooo/oooe////aae
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

To date, computational models have been applied to some extent for the analysis of nearly all medical imaging modalities and for a wide variety of clinical tasks [2]. Since it is critical to ensure patient safety and system effectiveness in healthcare applications, rigorous and thorough testing procedures must be performed in order to study performance in the intended population including subpopulations of interest. To prevent estimates that might be biased by overfitting, model testing is typically performed on a previously unseen dataset. However, datasets consisting of patient images may present limited distribution of the variability in human anatomy and may not always capture rare, but life-critical cases, and may be biased towards specific populations or parameters of image acquisition devices dominant at specific clinical sites. In addition, patient data and associated health records may not be available due to patient privacy, cost, or additional risk associated with additional imaging procedures. Precise mass location and extent (e.g., mass boundaries) are typically not available in the patient's records, and it is burdensome, error-prone, and sometimes impossible to collect this information retrospectively. In many medical imaging applications, these limitations pose a significant barrier to development and evaluation of novel computational techniques in medical imaging products.
We propose evaluating AI models using physics-based simulations. We create realistic test cases by imaging digital objects using digital image acquisition systems. Our in silico testing pipeline offers the ability to control both object and acquisition parameters, and generate highly realistic test cases (see Figure 1). We show that digital objects and computer simulated replicas of image acquisition devices offer a rich source of realistic data capturing a variety of patient and imaging conditions for evaluation purposes. In particular, our approach (and associated dataset) allows for performing comparative analysis of Al performance across physical breast properties (e.g., mass size) and imaging characteristics (e.g., radiation dose). Such testing typically cannot be performed with patient data, as the data may be too costly to collect or unsafe to acquire (e.g.. one cannot ethically re-image the same patient multiple times using ionizing radiation). Our contributions in this work can be summarized as follows:
We demonstrate that, using this approach, we can detect differences in Al model performance based on selected image acquisition device or physical object model parameters. Specifically, we evaluate the effect of image acquisition (radiation dose) and object model (breast and mass densities, mass size) parameters on the performance of the Al model.
We release a dataset, M-SYNTH, to facilitate testing with pre-computed data using the proposed pipeline. The dataset consists of 1,200 stochastic knowledge-baad badedd models and their associated digital mammography (DM) images with varying physical (breast density, mass size and density) and imaging (dose) characteristics.
2 Background
First, we introduce the concepts of knowledge-based models and physics-based imaging simulation that form the in silico imaging pipeline, the foundation of our work.
Object Models. Knowledge-basse (KB) models incorporate information about the physical world into the data generation process to create realistic virtual representations of human parts or organs [3]. As discussed in [1], large cohorts of digital stochastic human models can be represented by:
(f,) = Σ (r),,
(I)
where 8 denotes a particular state or random realization of a digital human in a cohort of size S, r denotes a spatial variable, on denote expansion (basis) functions, and 0, denote expansion coefficients. Knowledge-bard models specifically are constructed by sampling a set of 0, in Eq. I from distributions representing the relevant model characteristics, given specific on based on the application. The characteristics of the distributions are often derived from physical or biological measurements. In the case of breast, knowledge-basse based models allow to vary physical patient characteristics including breast size, breast shape, mass size and mass density (see Figures 2, 3 and 4).
Specifically, the object (breast) is a model D, parameterized by vector J characterizing a fixed, user- defined set of physiological properties (e.g., breast density, mass presence, mass size, glandularity). Given a sample it,, we can generate realistic, high-resolution object Ss = D(r.). We rely on Graff's s breast model [3] as the KB model for this project and describe its properties in Section 3.

---

Digital Mammography (DM) image generation. Once created. KB models are imaged using simulations of x-ray transport through the materials present in each KB model. The image acquisition device I is parametric model that receives the object d, as well as user-defined choices for control parameters y (e.g., detector type, radiation dose) and outputs an image ri.j = [(d;, Ut) given a sample choice of parameters Uj and an input object d,. Parameters of such system (e.g., geometry, source characteristics, detector technology. anti-scatter grid, etc.) can emulate system geometries and x-ray acquisition parameters found in commercially available imaging device (e.g., mammography) specifications. In our work, we used MC-GPU [4], a Monte Carlo x-ray simulation software implemented on GPUs that generates mammography images. Additional details for this component of the pipeline can be found in Section 3.
Related work in generative image models. The in silico imaging pipeline described above highly related to medical imaging generation using generative models. One popular type of generative model is a generative adversarial network (GAN) [5], which learns mapping from a low-dimensional representation to images at resolution. Generative models have been applied to a variety of medical image generation tasks [6]. For example, Guan [7] showed that GAN-generated synthetic images can be used to augment a smaller patient breast image dataset for breast image classification. [8] introduced image- GAN to generate high resolution images conditioned on pixel-level mask constraints. GANs may not correctly capture the link between input parameters and outputs, and thus, prone to generating unrealistic examples [9]. A number of alternative types of generative models [10, 11, 12, 13, 14] have been developed that address its limitations, such as training instabilities and unrealistic output images. A key advantage of generative models is that their run time can be faster than fully-detailed, object-space simulations, and it remains important to explore and compare both techniques. Their key limitation is that they require large training datasets and typically learn noise and artifacts from the imaging system [15]. In particular, all image acquisition systems have null space, i.e., the set of object-space details that are not observed in the acquired images due to imaging system limitations (e.g., finite spatial and temporal resolution). Null space constraints limit the ability of generative models to describe certain components of patient anatomy and pathology. Simulation-
testing has been proposed in other fields, such as autonomous vehicle navigation [16], and is related to the concept of generating adversarial perturbations in the image [17, 18, 19] and the physical property space [20, 21, 22]. For example, [23] introduced 3DB, a phato-realistic simulation framework debug and improve computer vision models. Inspired by these works, we propose to evaluate medical imaging AI using images generated using KB models and physics simulations and release a dataset facilitate such exploration.
3 Dataset Generation
The use of in silico imaging allows for the generation of large object and image datasets without the need of human clinical trials. Here, we take advantage of the benefits of the in silico approach to perform comparative analysis of AI model performance across different physical properties of the case population of breast models. We rely on the VICTRE pipeline for generating breast models and their corresponding DM images. Previous work [24] has shown that the VICTRE pipeline replicated the results of a clinical study comparing DM and digital breast tomosynthesis (DBT) involving hundreds of enrolled women. An overview of the data generation process can be seen in Figure 1.
Breast Model Synthesis. In silico breast models [3] (also known as breast imaging phantoms) were generated using a procedural analytic model which allows for adjusting various patient characteristics including breast shape, size and glandular density. The models are compressed in the craniocaudal direction using FeBio [25], an open source finite-element software. We simplified the breast materials in non-glandular (as fat) or glandular tissue with Young's modulus and Poisson ratio of E = Pa, 0.49 and E 15Pa, = 0.49, respectively. Lesions were inserted in subset to create the signal-present cohort. These models were then imaged using a state-of-the-ar Monte Carlo x-ray transport code (MC-GPU) [4].
We studied breast densities of extremely dense (referred as "dense"), heterogeneously dense (referred to as "hetero"), scattered, and fatty, matching the distributions from [24]. For each breast density, a different breast size is used to correspond with population statistics. Therefore, the dense breast is the smallest, followed by heterogeneously dense, then scattered, and then fatty, Each breast
'See VICTRE Github Page and FDA Regulatory Science Tools (RST) Catalog.
3

---

model was compressed to 3.5 cm, 4.5 cm, 5.5 cm, and 6.0 em for each respective density, mimicking the organ compression during the imaging. Random spiculated breast masses were generated using the de Sisternes model [26] with three different sizes (5 mm, 7 mm and 9 mm radii) and mass density was set to be a factor of glandular tissue density (1.0, 1.06 and 1.1 times). Note that for dense and hetero breasts, we only used mass sizes of 5 and 7 mm, since 9 mm masses do not fit within the breast region. No micro-calcification clusters were inserted. To create the signal-present cohort, a single spiculated mass was inserted in half of the cases at randomly chosen locations chosen from a list of candidate sites determined by the position of the terminal duct lobular units. The resulting in silico dataset comprises of 1 1,200 digital breast models, corresponding to 300 patients per breast size/density. Compared to the original VICTRE trial [24], we introduce variations in mass size and density. Samples of model realizations are shown in Figures 2, 3 and 4. Note that the bounding boxes are only to make the masses more conspicuous for visualization purposes only.
Digital Mammography (DM) Generation. To simulate the x-ray imaging process, we used MC-GPU [4]. a Monte Carlo x-ray simulation software implemented on GPUs that generates DM images. The detector model relies on system geometries and x-ray acquisition parameters inspired by the currently available Siemens Mammomat Inspiration DM system. The dosimetric and x-ray acquisition parameters were selected based on publicly available device specifications and clinical recommendations for each compressed breast thickness and glandularity. We applied 20-100% of the clinically recommended dose for each breast density. See Badal al. [4] for the exact parameter values and doses delivered to each breast and Sengupta et al. [28] for additional details. X-ray photons arriving at the detector are tracked until first photoelectric interaction incorporating fluorescence effects by generating and tracking a secondary x-ray based on the fluorescence yield in a uniformly random direction. Electronic noise is added to the pixel variance. The focal spot blurring in the source was modeled as a 3D Gaussian probability distribution with a wwwth--t-half--aaaaeee of 300 μum A tungsten anode filtered with 50 jam rhodium was used with a peak voltage of 28 kV for fatty and scattered breasts and 30 kV for dense and heterogencously dense breasts. The same analytical anti-scatter grid was also included for generating the DM images. (5:1 ratio, 31 line pairs/mm), see [4]. The resulting detector model (known as DIR in [28]) is representative of solid-state amorphous selenium transducer in direct detector configuration. Visualizations of generated images and masses can be seen in Figure 5. A summary of complete parameters used to generate data points in the presented dataset is described in Table 1. In Figure 7, we report statistics of dose levels corresponding to the dataset.
Parameter
Considered Values
Breast phantom density
Dense, Hetero, Scattered, Fatty 5, 7. 1.0, 1.06, 1.1
Mass radius (mm)
Mass density
Relative Dose
20%, 40%, 60%, 80%, 100% DIR
Detector type
Table 1: Parameters and their values corresponding to the M-SYNTH dataset.
4
Related Datasets
To date, number of datasets for mammographic image analysis have been collected (see Table 2).
The majority of datasets are created from patient data collected from DM [29, 30, 31, 32) or digital breast tomosynthesis (DBT) [33, 34) scans from various clinical sites. The DREAM Challenge [35] offered datasets for development of Al-based mammography analysis techniques. Patient datasets vary widely in the types of labels available, and the data may be biased toward the demographic characteristics of patients at the source site. While there exist datasets, such as the EMory BrEast imaging Dataset (EMBED) [34], that specifically focus on equal representation (in this case, equal representation of African American and White patients), collecting a truly balanced dataset across all possible characteristics may not be possible with patient cases.
We found only two in silico datasets for mammography analysis. The first dataset, published by Sarno [36], consists of 150 detient-derived digital breast models with uncompressed computational breast phantoms derived from 3D breast images acquired with an in-house dedicated breast computed tomography (CT) scanner. The models were processed by voxel classification algorithm into four

---

(a) 5 mm
(b) mm
(c) mm
(d)
(e)
(f)
Figure 2: Effect of varying mass size (5 mm to 9 mm radius) in fatty breast. Two breast models are shown, first: (a)-(c), and second: (d)-(f). Dose (# of hist.) 2.22 x 10 and mass density 1.1 remain constant. Bounding boxes placed here to indicate the location of the masses.
(a) 1.0
(b) 1.06
(c) 1.1
(d) 1.0
(e) 1.06
(I) 1.1
Figure 3: Effect of varying mass density (1.0 to 1.1 times glandular tissue density) in a fatty breast. Two models shown, first: (a)-(c), and second: (d)-(f), Dose (# of hist.) 2.22 x 10 and mass size 7 mm remain constant. Bounding boxes are placed here to indicate the location the masses.
Fatty
Hetero
Scottered
(a) Fatty
(b) Scattered
(c) Hetero
(d) Dense
(e) Renderings
Figure 4: Cohort variability. Varying breast density: R) Fatty, Scattered, Heterogeneously
dense, and Dense with mass size 7 mm and mass density 1.1. Note that dose changes with breast density. (e) shows artistic renderings of models for each composition (details Kim al. [27].
materials (air, adipose tissue, fibroglandular tissue, and skin). The second dataset the VICTRE [24] collection that consists of about 3,000 digital patients with breast sizes and densities representative of screening population. Digital microcalcification clusters and spiculated masses were inserted in the voxelized phantoms to create the positive cohort. The phantoms were imaged in silico to produce digital mammogram projections and digital breast tomosynthesis volumes. In comparison both of these datasets, our work contains more significant variability in breast and mass characteristics, as well as range applied dose levels for image acquisition, in order facilitate comparative evaluations of Al across characteristic changes.
5

---

(a) 4.44 10
(b) 8.88 10
(c) 10 (d) 1.78 10 (e) 2.22 10
Figure 5: Imaging. Effect of increasing imaging dose (# of hist.) from left to right. Mass size of 5 im and mass density of 1.1 remain constant.
% of Clinical Dose
80
30-
20+
30-
60
Fatty
Scattered
Heter ro Dansa
20
0.5
1.0
1.5
2.0
Dose [number histories] 1e10
Figure 6: Dose distribution (# of hist.) and percentages of optimal dose considered by breast density.
1.0
05 2.3
Figure 7: Glandular dose distributions for the dataset.
Real putlent datnwets
Dutaset
DM present
DET present
# cases
# Imngges
Image calegocies
Population
Thoke [T5]
No
22092
Cancerr henign, actimmble normal
USA
ADMANI 25]
Yea
No
629661
4411263
Nammal recall
Severa countracs
EMBED
Yes
[1600)
3383659
honderline, spggtive, high
USA'
CMMD [1]]
Ves
Benggn malignant
China
Ves
Beaign, mnpligattt, memua
Retugal
OPTMMAM [51]
Yes
172,382
172,282
Normal, interval cancer, benign, milignan
UK
In Silico dolasets
Dutaset
DM present
DBT present
images 150F
Image categuries
Phantom variability
Imaging
Samno [36]
Yes
Yes
Nermal
No
No
VICTRE
M-SYNTH (Our))
Nesptive pasliiv calort
sobtt avalaabe RSNA Cancer Deverion All challenge.
"N% availabie AWS: contains lesion.
reprscetation of Africnn and White 13 lesion dessties, lesion sizes, 4 beaat denaties, differen doecs
150 uncompresasd, compesseed images fous becast dessiies, lesiors azmus posiivv cobor:
A comoponding Dit imag desaet will be prosited in finm solewur desect.
Table 2: Summary of existing mammographic image datasets.
5 Results and Analysis
In this section, we present an approach to using our M-SYNTH dataset to evaluate an Al device. Formally, an image processing AI model F' takes as input an image and predicts a specific property of interest Frr) about the image. For example, such a model can predict the presence or absence of mass. Typically for Al models, their labels Ttrain = {(11, I2), (r2, 12).... (rr, [,)), and then evaluated on held-out dataset Teest- When using patient images, evaluation is limited to the variability contained in the samples and the annotations present across examples in the fixed test set Trest- Instead, we propose to generate Tirain and T Ttest dynamically using D and / described above in order to test across variations in model and acquisition parameters y.
a neural network and trained on a dataset of images and

---

5.1 Implementation Details
Evaluation Metries We evaluate performance using the area under curve (AUC) metric for mass detection task. Specifically, we treat evaluation multiple reader multiple case study, where an Al model is a single reader. Multiple readers are obtained by re-training the model with different random seeds. We rely the iMRMC software [37, 38] to identify associated confidence intervals.
Network Training We represent the Al-enabled device as a neural network with an efficientnet_ b0 architecture, receiving an image with one channel and dimensions of 224 by 224, and outputting binary mass presence label. The network is trained with batch size 64 using binary cross entropy loss (BCE) and optimized using RMSProp optimizer (with learning rate 0.0001). We rely on the timm library [39] and fine-tune the model pre-trained with ImageNet [40]. We also compared performance with alternative architectures supplementary material).
1ammcllpatcch6..11m 2_224442222222 and vgg_ 16), but results were very similar (see
For each specific breast density, radiation dose level, and mass size and density, the 300 images in the M-SYNTH dataset were divided into 200 for training. 50 for validation, and 50 for test. For comparison, we also train the Al device on 410 patient DM images from the INBreast dataset [32], where images were obtained using MammoNovation Siemens full-field digital mammography system with solid-state amorphous selenium detector. We use the same pre- prepprccsssing and training regimes on this dataset and learn a network to predict mass presence. The trained models on the real patient dataset were then tested 50 examples of M-SYNTH dataset for each specific breast density, dose level, and mass size and density. The full experimental setup is implemented Python and C over a cluster with 50 Tesla V100-SXM2 GPUs.
5.2 Experimental Results
We identify two tasks that can be performed using our method. In the subgroup analysis task, we train and test an model using the released synthetic (M-SYNTH) dataset to identify performance changes specified subgroups. In the putient data evaluation task, we study how an Al model trained on patient data (InBreast) performs on the proposed M-SYNTH dataset. This task can help identify where the trained model may show variable performance for different subgroups belonging to the target population.
1.0-
20
29
0.50
0.5
2.8
0.8
3.46
39
0.7
0.94
0.6-
Taenng men roder
0.6
0.90
2.4
To brenl oro cimut
35
3.00
192
2.0
3ite
LOS
110
24550
0.50
20
60
60
100
(a) Mass size
(b) Mass density
(c) Breast density
(d) Dose level
Figure 8: Subgroup analysis. Performance change across (a) mass size, (b) mass density, (c) breast density, and (d) radiation dose, for models trained and tested on our M-SYNTH dataset. These parameters remained constant for the set of experiments performed during both training and test: (a) Fatty breast phantom, mass density of 1.06, and relative dose 100%. (b) Fatty breast phantom, mass size of 7 mm, and relative dose of 100%. (c) Mass density of 1.06, mass size of 7 mm, and relative dose of 100%. (d) Fatty breast phantom, mass density of 1.06, and mass size of mm.
Subgroup Analysis. In Figures 8 and 9, report the results of the AI model performance at detecting masses, when the model is trained and tested on the our dataset (see Section 5.1 for details of splits). We find that masses with larger sizes or higher densities (Figures 8a-b) are more easily detected. Although models trained on all sizes or mass densities have the highest performance, when models are trained on smaller masses or lower densities, they generalize better to other masses (more difficult cases). The performance of the models are highest when they are tested and trained on the same breast density and decrease as the density the test breast phantom differs from the train phantom (Figures 8c), The dose levels applied in this study have minimal impact the performance
7

---

of the models and resulted in similar AUC values (Figures 8d). Evaluation of the performance change across all the breast densities (Figures 9a-b) reveals that the AUC improves with larger mass density and mass size, yet is impacted by the breast density, where mass detection performance is lowest in high-density breasts (dense) and highest in low-density breasts (fatty) in most of the cases, consistent with findings from clinical practice.
"Iot broasi dena fally
Test brezst persty: scuttoredd
Test breast ders hetaro
Tiss brcust clanaity: derse
1.0
1,2
1.0
2.0
2.8
0.5
2.5
0.5
0.7
3.7
6.7
2.-
0.8
0.5
0.6
lest ras denilly 1.5
Tor mas desiity 1.0
Test ras denict 1.5 Telt rass drecit 1.06
Test restirasi radiddadey 1.0
3.5
list mas denatty 1.8s
0.5
Tolt mas desiity 1.04
3.5
0.5
deritry
mass (mm]]
[mm]
[mon]
(a) AUC as function of mass size across all breast densities.
"kst breast dena farty
creadt cersity: ty: acattored
breast dens by: hetero
Test breast cematty: derse
1.0
1.0
3.0
1.0
3.8
0.8
9.5
0.6
0.7
0.7
0.7
0.7
9.6
0.6
2.5
0.5
2.5
0.5
1.00
3.02
2.56
1:50
1.50
1.07
3.06
3.50
3.00
1.07
1.44
1.05
3:50
3.50
1.22
1.s
nass Borsky
donaity
denicty
masi sonsiyy
(b) AUC as function of mass density across all breast densities.
Figure 9; Subgroup analysis. Performance changes for models trained and tested our M-SYNTH dataset. For each data point, the model is trained on 250 images with masses of radii of 7 mm and mass densities of 1.06, and tested on 50 images with mass characteristics shown in plots for each specific breast density. The radiation dose level remains constant at 100% of the clinically recommended dose for each breast density during training and test.
Patient Data Evaluation. In Figure 10, we report experiments where the AI model is trained on INBreast data and evaluated on the M-SYNTH data. Although the performance results for all experiments are lower in general, we find similar set of trends when the model is trained on M-SYNTH data. Note that we have made no attempt to match the radiation dose levels or the image acquisition parameters for these comparisons using patient images. Even though the simulated pipeline is designed to replicate a specific DM system with a particular detector technology and technique factors, the comparison suggests similarity between the datasets. The images are qualitatively different but overall have similar glandular patterns which is an important consideration for the realism of the task of detecting masses in a noisy background. We also assessed similarity between INBreast and M-SYNTH datasets in terms of low-level pixel distributions using first five statistical moments: mean, variance, skewness, kurtosis, and hyperskewness. We found that there is a reasonably good alignment in terms of moments, especially when the synthetic images were included at all four breast densities (see supplementary material). Future work should develop more detailed comparison including radiomics features for the training and testing datasets used in the study to complement the validation of our approach.
Limitations. There are a number of limitations to our work. First, simulations may require long runtimes and demand large computational resources, thus somewhat limiting the amounts of data that can be generated. This limitation needs to be considered with respect to the difficulty of obtaining large patient image datasets with known mass locations. In addition, data can be pre-generated offline (as we do with the M-SYNTH dataset), therefore, removing the large runtime limit and computational burden off the user. Second, testing with simulations constrained to the variability captured by the parameter space of the object models for anatomy and pathology and the acquisition system. Thus, the complexity of the object model and acquisition system may need to be adjusted depending on the complexity of the questions to be investigated with simulated testing. In particular, a potential risk of testing using simulated data is missing the variability observed in patient populations. Finally, there is risk of misjudging model performance due to a domain gap between real and synthetic
8

---

Text treast dets farly
cread ceraty: acuitree
Text breast denaly: heforo
breant conalty: detee
1.0
Tnnt mass cenety: 1.0
1.0
The mai denety
2.2
The mart Sensl:
1.0
That mass cenety:
sensity
dessiy:
les: serdtt
Test
9.9
sendlt
0.9
desiity
9.5
0.9
ceniity.
0.7
0.7
9.7
0.7
16
0.5
0.8
0.6
85
9.5
24
0.4
2.4
0.4
Test masssee Inin]
m156 size (m/m)
mass size [nm]
Test mass Aec [mm]
(a) AUC as function of mass size across all breast densities.
Teit troaet denety teely
That bread denely: cratered
Teat browdl denely: hetern
The breatt cendly detee
Toil mask adz: 5.0 mm
1.2
Test masi alaz 3.2 mm
2.2
Taa: mase auzz: m.
2.0
Taat masa aic 5. mn
3.8
sau mm
adz: mm
0.9
Tost musi slaz mm
Tost mass aaaz 9.0 mm
3.5
Tax: mase adai 7.0mm
0.0
Test masa sic 7:2 mn
3.7
0.7
0.7
2.8
0.8
0.5
0.6
34
1:50
1.01
1:66
1:00
1.00
0.4
1.50
1.02
1.01
1.06
1.5
1.50
2.4
3.50
1.05
1.00
3.50
0.4
1.0
1.22
1.01
1
1000
3.10
Tist masa duralty
lost mass denulty
Test mass dena
Text maxi cursity
(b) AUC as function of mass density across all breast densities.
Figure 10: Model Evaluation. Performance changes for a model trained on 410 real patient images (INBreast dataset) and tested on our M- dataset. The test sets consist of 50 images using parameters shown in the plots. The test radiation dose is set to 100% of the clinically recommended dose for each breast density.
examples. However, the realism and sophistication of object- based modeling of the imaging pipeline is improving rapidly and may soon compete with other approaches, making approaches based on synthetic data useful and practical for regulatory evaluation of Al-enabled medical devices.
6 Conclusion and Future Work
We introduce and discuss an approach for validating AI models using physics-based simulations of digital humans from the object space to the image data, specifically for the task of breast cancer mass detection. The simulated images are highly realistic and offer a challenging test case for AI model evaluation. Our findings model performance increases with mass size and mass density as expected. Finally, we show that our approach can be used to validate model trained independent patient data. This finding suggests that the proposed simulation setup can be used as a framework for more general evaluation of medical AI devices. The goal of this study is to demonstrate as proof-of-concept the feasibility of using simulated data to evaluate the comparative performance of Al models. In future work, it would be important to assess the evaluation approach for additional parameters in terms of the distribution of the population of digital humans in the object space, and for a range of image acquisition systems (c.g.. by considering alternative simulators). By imaging a more diverse population of breast models, we hope to identify additional insights regarding AI evaluation. Finally, it is important to note that the testing is limited to the variability captured in the digital representations and may not fully indicate absolute real-world performance or trends. This study illustrates that physics-basee mammography images can represent less burdensome and cost-efficient approach for the evaluation of AI model performance across wide range of scenarios, including a variety of image acquisition parameters and diverse populations that may not be available or are hard to obtain from human studies. Moreover, this approach offers complementary evaluation paradigm that does not depend on the availability of patient data.
consistent with expected performance and show that the AI
simulation of
Acknowledgements
We thank Andreu Badal (OSEL/CDRH/FFDA) and anonymous reviewers for helpful suggestions, Kenny Cha, Mike Mikailov and the OpenHPC team (OSHL/CDDHH/DA)) for providing help with experiments, Akhonda, Mohammad (OSEL/CDRH/FDAA) for help with data release, and Andrea Kim

---

(OSEL/CDRH/FDA) for rendering visualizations of the 3D breast model. This is contribution of the US Food and Drug Administration and is not subject to copyright. The mention of commercial products herein is not to be construed as either an actual or implied endorsement of such products by the Department of Health and Human Services.
References
[1] A Badano, M Lago, E Sizikova, JG Delfino, S Guan, MA Anastasio, and B Sahiner. The stochastic digital human is now enrolling for in silico imaging trials-methods and tools for generating digital cohorts. arXiv preprint arXi::2301.0877 2023.
[2] Ana Barragán-Montent Umair Javaid, Gilmer Valdés, Dan Nguyen, Paul Desbordes, Benoit Macq, Siri Willems, Liesbeth Vandewinckele, Mats Holmström, Fredrik Löfman, et al. Artificial intelligence and machine learning for medical imaging: A technology review. Physica Medica, 83:242-256, 2021.
[3] Christian G Graff. A new, open-source, multi-modalie digital breast phantom. In Medical Imaging 2016: Physics Medical Imaging, volume 9783, pages 22-81. SPIE, 2016.
[4] Andreu Badal, Diksha Sharma, Christian G Graff, Rongping Zeng, and Aldo Badano. Mammography and breast tomosynthesis simulator for virtual clinical trials. Computer Physics Communications, 261:107779, 2021.
[5] lan Goodfellow, Jcan Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[6] Youssef Skandarani, Pierre-Mare Jodoin, and Alain Lalande. Gans for medical image synthesis: An empirical study. arXiv preprint arXiv:2105.055 2021.
Shuyue Guan and Murray Loew. Breast cancer detection using synthetic mammograms from generative adversarial networks convolutional neural networks. Journal of Medical Imaging, 6(3)00344 2019.
[8] Yinhao Ren, Zhe Zhu, Yingzhou Li, Dehan Kong, Rui Hou, Grimm, Jeffery R Marks, and Joseph Lo. Mask embedding for realistic high-resolution medical image synthesis. In MICCAI, pages 422-430. Springer, 2019.
[9] Elena Sizikova, Xu Cao, Ashia Lewis, Kenny Moise, and Megan Coffee. Improving computed tomography (ct) reconstruction via 3d shape induction. arXiv preprint arXiv:2208. 10937, 2022.
[10] Diederik Kingma and Max Welling, entodenn variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[11] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[12] Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint arxiv 1707.0576, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[14] Florincl-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey, arXiv preprint arXiv:2209.0477 2022.
[15] Weimin Zhou, Sayantan Bhadra, Frank Brooks, Hua Li, and Mark Anastasio. Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks. Journal of Medical Imaging, 91):015503, 2022.
[16] Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In CVPR, pages 9909 9918,9918111 2021.
[17] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey, leee Access, 6:14410- 14430, 2018.
[18] Samuel G Finlayson. John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane. Adversarial attacks medical machine learning. Science, 363(6433):1287-122 -12891 2019.
10

---

[19] Hokuto Hirano, Akinori Minagi, and Kazuhiro Takemoto. Universal adversarial attacks on deep neural networks for medical image classification. BMC medical imaging, 21:1-13, 2021.
[20] Chaowei Xiao, Dawei Yang, Bo Li. Jia Deng. and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In CVPR, pages (898-6907, 2019.
[21]
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang. Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan I. Yuille. Adversarial attacks beyond the image space. In CVPR, pages 4302-4311, 2019.
[22] Hsuch-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alee Jacobson. Beyond pixel norm-balle Parametric adversaries using an analytically diffcrentiable renderer. arXiv preprint arXiv: 1808 1808.0261,, 2018.
[23] Guillaume Leclerc, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang. et al. 3db: A framework for debugging computer vision models. arXiv preprint arXiv:2106.03800 03805, 2021.
[24] Aldo Badano, Christian G Graff, Andreu Badal, Diksha Sharma, Rongping Zeng, Frank W Samuelson, Stephen J Glick, and Kyle J Myers, Evaluation of digital breast tomosynthesis replacement of full-field digital mammography using an in silico imaging trial. JAMA network open, 1())x1185744-c18555 11 2018.
[25] Steve A. Maas, Benjamin J. Ellis, Gerard A. Ateshian, and Jeffrey A. Weiss. FEBio: Finite Elements for Biomechanics. Journal of Biomechanical Engineering, 134(1);01 1005, 02 2012.
[26] Luis de Sistemns, Jovan G Brankov, Adam M Zysk, Robert A Schmidt, Robert M Nishikawa, and Miles N Wernick. A computational model to generate simulated three-dimensional breast masses. Medical physicx, 42(2):1098-1113 2015.
[27] Andrea Kim, Aunnasha Sengupta, and Aldo Badano. Automated animation pipeline for visualizing in silico tumor growth models. Physics of Medical Inging, 12463, 2023.
[28] Aunnasha Sengupta, Andreu Badal, Andrey Makeev, and Aldo Badano. Computational models of direct and indirect x-ray breast imaging detectors for in silico trials. Medical Physics, 49(1 1):6866-6870, 2022.
[29] Helen ML Frazer, Jennifer SN Tang, Michael S Elliott, Katrina M Kunicki, Brendan Hill, Ravishankar Karthik, Chun Fung Kwok, Carlos A Peña- Aeia--Solozzaoo, Yuanhong Chen, Chong Wang, et al. Admani: Annotated digital mammograms and associated non- non-image datasets. Radiology: Artificial Intelligence. 5(2)x220072, 2022.
[30] Chunyan Cui, Li Li. Hongmin Cai, Zhihao Fan, Ling Zhang, Tingting Dan, Jiao Li, and Jinghua Wang- The chinese mammography database (cmmd); An online mammography database with biopsy confirmed types for machine diagnosis of breast. Data Cancer Imaging Arch, 2021.
[31] Mark D. Halling-Browe Brown, Lucy M. Warren, Dominic Ward, Emma Lewis, Alistair Mackenzie, Matthew Wallis, Louise S. Wilkinson, Rosalind M. Given-Willon Rita McAvinchey, and Kenneth C. Young. Optimam mammography image database: A large-scale resource of mammography images and clinical data, Radiology: Artificial Intelligence, 3(1), 2020.
[32] Ines C Moreira, Igor Amaral, Inés Domingues, Autónio Cardoso, Maria Joao Cardoso, and Jaime S Cardoso. Inbreast: toward a flll-iied digital mammographic database. Academic radiology. 19(2):23-244) 2012.
[33] Mateusz Buda, Ashirbani Saha, Ruth Walsh, Sujata Ghate, Nianyi Li, Albert Šwięcicki. Joseph Y Lo, and Macicj A Mazurowski. Dete ction of masses and architectural distortions in digital breast tomosynthesis: publicly available dataset of 5,060 patients and a deep learning model. arXiv preprint 1XX092001..07995 2020.
[34] Jiwoong Jeong. Brianna L Vey, Ananth Bhimireddy, Thomas Kim, Thiago Santos, Ramon Correa, Raman Dutt, Marina Mosunjac, Gabriela Oprea-llies, Geoffrey Smith, et al. The emory breast imaging dataset (embed): A racially diverse, granular dataset of 3.4 million screening and diagnostic mammographic images. Radiology: Artificial Intelligence, 5(1) 5(1):c220047, 2023.
[35] Berkman Sahiner. Digital mammography dream challenge overview (conference presentation). In Medical Imaging 2017: Computer-Aided Diagnosis, volume 101 14,4, pages 1159-1159 SPIE, 2017.
[36] Antonio Samo, Giovanni Mettivier, Francesca di Franco, Antonio Varallo, Kristina Bliznakova, Andrew M. Hernandez, John M. Boone, and Paolo Russo. Dataset of -etient-derived digital breast phantoms for in silico studies in breast computed tomography, digital breast tomosynthesis, and digital mammography. Medical Physics, 48(5):6682 2693, 2021.
11

---

[37] Brandon D. Gallas, Andriy Bandos, Frank W. Samuelson, and Robert F. Wagner. A framework for random-effects analysis: Biases with the bootstrap and other variance estimators, Communications Statisticy Theory and Methods, (81 2009.
[38] RST Catalog, iMRMC: Software for the Statistical Analysis of multi-reader multi-case studies, June 2022.
[39] Ross Wightman. Pytorch image models. httpa://gitthbb.cmm/// http:///////aooaaa/o/o/oaaaaaaaa 2019.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database, In CVPR. pages 248-255 leee, 2009.
12


Reviews:
[{"review_id": "3HOUaoQQ9s", "reviewer": "Reviewer_gzvQ", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations and risks of their work (cf. paragraph in 5.2). I agree with the authors that the largest risk is that there potentially exist additional variations in patient populations than the ones studied in this work. An additional risk is that the images are not entirely realistic and therefore represent a dataset shift from real data which might cause machine learning model performance to drop significantly.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The work contributes to creating benchmarks in the field of medical imaging where first machine learning models have recently been certified  as computer-aided diagnosis tool. Medical image datasets often lie in the low-data regime while at the same time containing highly heterogeneous samples. This heterogeneity is caused by multiple factors: general biological variability (here: of breast tissue), different expressions of the pathology (here: size and density of masses) and image acquisition. The suggested approach allows to vary along those dimensions and therefore cover relevant cases which might be rare and therefore difficult to collect as clinical data. While it should not replace current clinical validations, it is a valuable addition to discover potential model flaws and test for more rare edge cases. Furthermore, the authors also investigate the effect of predictive performance on the whole spectrum of data when training on only one of the subgroups. This nicely demonstrates that one should keep training data statistics in mind when making statements about generalization. The experiments are carried out in a clear and logical outline and cover all relevant aspects. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DNLiQbidvI", "reviewer": "Reviewer_6Vc7", "summary": "", "questions": "", "limitations": "Please refer to the above points.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper proposed an evaluation pipeline for  testing an AI model for mammography using synthetic data generated by in silico methods\n- The paper introduces a novel dataset of synthetic mammograms, M-SYNTH, that covers a range of breast and mass characteristics and dose levels that are not easily available in patient data.\n- The paper demonstrates the use of in silico methods for generating realistic and diverse data for testing AI models and shows how the physical and imaging parameters can influence the model performance.\n- The paper provides a comparison of the model performance on synthetic and patient data and shows some agreement between the two datasets, suggesting the validity of the synthetic data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ojlxIPdo4L", "reviewer": "Reviewer_ctVH", "summary": "", "questions": "", "limitations": "The authors are well aware (section 5, 6) that simulated data may not cover all the variations seen in the wild. But they may be over-claiming when saying this dataset could be used as a framework to test medical AI devices, as it's not clear as to how representative of the universe of real data, this simulated dataset is. I think that it should be clarified that this dataset should be used mostly for comparative analysis (unless I have misunderstood of course).", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is clearly written and well structured. \n\nThe paper implements a physics-based method rather than others such as GANs. Mammography and X-ray in general are widely studied, and reliable simulators are well accepted. The pipeline takes the Graff phantom [3], re-uses the VICTRE pipeline [24], and an x-ray simulation process [4]. These have all been previously reported on and accepted in the literature. So, the methodology is well-principled.\n\nThe resulting dataset provides a balanced set of 4 densities, 3 radii, 3 tumour densities and 5 relative doses, so covers a reasonable clinical range, in a methodical fashion.\n\nIn the medical field, datasets are frequently much smaller than in computer vision, and harder to come by for a variety of privacy and regulatory reasons. As such, this dataset is very relevant to the wider medical image community, and would be valuable for research. \n\nThe use of simulations circumvents many of the ethical problems, and the authors are clearly aware of the limitations of simulated datasets and present their work honestly (i.e. realistically).\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviewers are generally positive in recommending the acceptance of this manuscript but also raise concerns such as more evaluation metrics and more experiments on advanced models.
Decision: Accept (Poster)