* Discriminator networks (in yellow) either force the encoder to take temporal progression into account ($D_z$) or try to determine if the output of the generator are real or generated images ($D_b$).

* Biological constraints (in grey) force the previous generated image of the same participant to be less atrophied than the next one (voxel loss) and learn to find the diagnosis thanks to regions of the generated images (regional loss).

* Profile weight functions (in blue) aim at finding appropriate weights for each loss to compute the total loss.

The assembly of all these components allows learning a longitudinal model that characterizes the progression of the atrophy of each region of the brain. This atrophy evolution can then be visualized thanks to a neurodegeneration simulation generated by the trained model by sampling missing intermediate values.

## 3.7 Benchmarks Conducted in the Literature

This section describes studies that compared several interpretability methods. We separated evaluations based on metrics from those which are purely qualitative. Indeed, even if the interpretability metrics are not mature yet, it is essential to try to measure quantitatively the difference between methods rather than to only rely on human perception, which may be biased.

### 3.7.1 Quantitative Evaluations

Eitel and Ritter [37] tested the robustness of four methods: standard perturbation, gradient input, guided back-propagation, and LRP. To evaluate these methods, the authors trained ten times the same model with a random initialization and generated attribution maps for each of the ten runs. For each method, they exhibited significant differences between the averaged true positives/negatives attribution maps of the ten runs. To quantify this variance, they computed the $l_2$-norm between the attribution maps and determined for each model the brain regions with the highest attribution. They concluded that LRP and guided back-propagation were the most consistent methods, both in terms of distance between attribution maps and most relevant brain regions. However, this study makes a strong assumption: to draw these conclusions, the network should provide stable interpretations across retrainings. Unfortunately, Thibeau-Sutre et al. [51] showed that the study of the robustness of the interpretability method and of the network should be done separately, as their network retraining was not robust. Indeed, they first showed that the interpretability method they chose (optimized perturbation) was robust according to different criteria, and then they observed that network retraining led to different attribution maps. The robustness of an interpretability method thus cannot be assessed from the protocol described in [37]. Moreover, the fact that guided back-propagation
is one of the most stable method meets the results of [6], who observed that guided back-propagation always gave the same result independently from the weights learned by a network (see Subheading 4.1).

Böhle et al. [34] measured the benefit of LRP with β-rule compared to guided back-propagation by comparing the intensities of the mean attribution map of demented patients and the one of cognitively normal controls. They concluded that LRP allowed a stronger distinction between these two classes than guided back-propagation, as there was a greater difference between the mean maps for LRP. Moreover, they found a stronger correlation between the intensities of the LRP attribution map in the hippocampus and the hippocampal volume than for guided back-propagation. But as [6] demonstrated that guided back-propagation has serious flaws, it does not allow drawing strong conclusions.

Nigri et al. [45] compared the standard perturbation method to a swap test (see Subheading 3.4) using two properties: the continuity and the sensitivity. The continuity property is verified if two similar input images have similar explanations. The sensitivity property affirms that the most salient areas in an explanation map should have the greater impact in the prediction when removed. The authors carried out experiments with several types of models, and both properties were consistently verified for the swap test, while the standard perturbation method showed a significant absence of continuity and no conclusive fidelity values [45].

Finally, Rieke et al. [48] compared four visualization methods: standard back-propagation, guided back-propagation, standard perturbation, and brain area perturbation. They computed the Euclidean distance between the mean attribution maps of the same class for two different methods and observed that both gradient methods were close, whereas brain area perturbation was different from all others. They concluded that as interpretability methods lead to different attribution maps, one should compare the results of available methods and not trust only one attribution map.

### 3.7.2 Qualitative Evaluations

Some works compared interpretability methods using a purely qualitative evaluation.

First, Eitel et al. [38] generated attribution maps using the LRP and gradientinput methods and obtained very similar results. This could be expected as it was shown that there is a strong link between LRP and gradientinput (see Subheading 2.3.2).

Dyrba et al. [36] compared DeconvNet, guided back-propagation, deep Taylor decomposition, gradientinput, LRP (with various rules), and Grad-CAM. The different methods roughly exhibited the same highlighted regions but with a
significant variability in focus, scatter, and smoothness, especially for the Grad-CAM method. These conclusions were derived from a visual analysis. According to the authors, LRP and deep Taylor decomposition delivered the most promising results with a highest focus and less scatter [36].

Tang et al. [49] compared two interpretability methods that seemed to have different properties: guided Grad-CAM would provide a fine-grained view of feature salience, whereas standard perturbation highlights the interplay of features among classes. A similar conclusion was drawn by Rieke et al. [48].

### 3.7.3 Conclusions from the Benchmarks

The most extensively compared method is LRP, and each time it has been shown to be the best method compared to others. However, its equivalence with gradientinput for networks using ReLU activations still questions the usefulness of the method, as gradientinput is much easier to implement. Moreover, the studies reaching this conclusion are not very insightful: [37] may suffer from methodological biases; [34] compared LRP only to guided back-propagation, which was shown to be irrelevant [6]; and [36] only performed a qualitative assessment.

As proposed in conclusion by Rieke et al. [48], a good way to assess the quality of interpretability methods could be to produce some form of ground truth for the attribution maps, for example, by implementing simulation models that control for the level of separability or location of differences.

# 4 Limitations and Recommendations

Many methods have been proposed for interpretation of deep learning models. The field is not mature yet, and none of them has become a standard. Moreover, a large panel of studies has been applied to neuroimaging data, but the value of the results obtained from the interpretability methods is often still not clear. Furthermore, many applications suffer from methodological issues, making their results (partly) irrelevant. In spite of this, we believe that using interpretability methods is highly useful, in particular to spot cases where the model exploits biases in the data set.

## 4.1 Limitations of the Methods

It is not often clear whether the interpretability methods really highlight features relevant to the algorithm they interpret. This way, Adebayo et al. [6] showed that the attribution maps produced by some interpretability methods (guided back-propagation and guided Grad-CAM) may not be correlated at all with the weights learned by the network during its training procedure. They prove it with a simple test called “cascading randomization.” In this test, the weights of a network trained on natural images are randomized layer per layer, until the network is fully randomized. At each step,
they produce an attribution map with a set of interpretability methods to compare it to the original ones (attribution maps produced without randomization). In the case of guided back-propagation and guided Grad-CAM, all attribution maps were identical, which means that the results of these methods were independent of the training procedure.

Unfortunately, this type of failures does not only affect interpretability methods but also the metrics designed to evaluate their reliability, which makes the problem even more complex. Tomsett et al. [52] investigated this issue by evaluating interpretability metrics with three properties:

* **Inter-rater interpretability** assesses whether a metric always rank different interpretability methods in the same way for different samples in the data set.

* **Inter-method reliability** checks that the scores given by a metric on each saliency method fluctuate in the same way between images.

* **Internal consistency** evaluates if different metrics measuring the same property (e.g., fidelity) produce correlated scores on a set of attribution maps.

They concluded that the investigated metrics were not reliable, though it is difficult to know the origin of this unreliability due to the tight coupling of model, interpretability method, and metric.

**4.2 Methodological Advice**

Using interpretability methods is more and more common in medical research. Even though this field is not yet mature and the methods have limitations, we believe that using an interpretability method is usually a good thing because it may spot cases where the model took decisions from irrelevant features. However, there are methodological pitfalls to avoid and good practices to adopt to make a fair and sound analysis of your results.

You should first clearly state in your paper which interpretability method you use as there exist several variants for most of the methods (see Subheading 2), and its parameters should be clearly specified. Implementation details may also be important: for the Grad-CAM method, attribution maps can be computed at various levels in the network; for a perturbation method, the size and the nature of the perturbation greatly influence the result. The data on which methods are applied should also be made explicit: for a classification task, results may be completely different if samples are true positives or true negatives, or if they are taken from the train or test sets.

Taking a step back from the interpretability method and especially attribution maps is fundamental as they present several limitations [34]. First, there is no ground truth for such maps, which are usually visually assessed by authors. Comparing obtained results
with the machine learning literature is a good first step, but be
aware that you will most of the time find a paper to support your
findings, so we suggest to look at established clinical references.
Second, attribution maps are usually sensitive to the interpretability
method, its parameters (e.g., β for LRP), but also to the final scale
used to display maps. A slight change in one of these variables may
significantly impact the interpretation. Third, an attribution map is
a way to measure the impact of pixels on the prediction of a given
model, but it does not provide underlying reasons (e.g., pathologi-
cal shape) or explain potential interactions between pixels. A given
pixel might have a low attribution when considered on its own but
have a huge impact on the prediction when combined with another.
Fourth, the quality of a map strongly depends on the performance
of the associated model. Indeed, low-performance models are more
likely to use wrong features. However, even in this case, attribution
maps may be leveraged, e.g., to determine if the model effectively
relies on irrelevant features (such as visual artefacts) or if there are
biases in the data set [53].

One must also be very careful when trying to establish new
medical findings using model interpretations, as we do not always
know how the interpretability methods react when applied to cor-
related features. Then even if a feature seems to have no interest for
a model, this does not mean that it is not useful in the study of the
disease (e.g., a model may not use information from the frontal
lobe when diagnosing Alzheimer's disease dementia, but this does
not mean that this region is not affected by the disease).

Finally, we suggest implementing different interpretability
methods to obtain complementary insights from attribution
maps. For instance, using LRP in addition to the standard back-
propagation method provides a different type of information, as
standard back-propagation gives the sensibility of the output with
respect to the input, while LRP shows the contribution of each
input feature to the output. Moreover, using several metrics allows
a quantitative comparison between them using interpretability
metrics (see Subheading 2.7).

4.3 Which Method
Should I Choose?

We conclude this section on how to choose an interpretability
method. Some benchmarks were conducted to assess the properties
of some interpretability methods compared to others (see Subhead-
ing 3.7). Though these are good initiatives, there are still not
enough studies (and some of them suffer from methodological
flaws) to draw solid conclusions. This is why we give in this section
some practical advice to the reader to choose an interpretability
method based on more general concepts.

Before implementing an interpretability method, we suggest
reviewing the following points to help you choose carefully.
* **Implementation complexity** Some methods are more difficult to implement than others and may require substantial coding efforts. However, many of them have already been implemented in libraries or GitHub repositories (e.g., [54]), so we suggest looking online before trying to re-implement them. This is especially true for model-agnostic methods, such as LIME, SHAP, or perturbations, for which no modification of your model is required. For model-specific methods, such as back-propagation ones, the implementation will depend on the model, but if its structure is a common one (e.g., regular CNN with feature extraction followed by a classifier), it is also very likely that an adequate implementation is already available (e.g., Grad-CAM on CNN in [54]).

* **Time cost** Computation time greatly differs from one method to another, especially when input data is heavy. For instance, perturbing high dimension images is time expensive, and it would be much faster to use standard back-propagation.

* **Method parameters** The number of parameters to set varies between methods, and their choice may greatly influence the result. For instance, the patch size, the step size (distance between two patches), as well as the type of perturbation (e.g., white patches or blurry patches) must be chosen for the standard perturbation method, while the standard back-propagation does not need any parameter. Thus, without prior knowledge on the interpretability results, methods with no or only a few parameters are a good option.

* **Literature** Finally, our last piece of advice is to look into the literature to determine the methods that have commonly been used in your domain of study. A highly used method does not guarantee its quality (e.g., guided back-propagation [6]), but it is usually a good first try.

To sum up, we suggest that you choose (or at least begin with) an interpretability method that is easy to implement, time efficient, with no parameters (or only a few) to tune, and commonly used. In the context of brain image analysis, we suggest using the standard back-propagation or Grad-CAM methods. Before using a method you do not know well, you should check that other studies did not show that this method is not relevant (which is the case for guided back-propagation or guided Grad-CAM) or that it is not equivalent to another method (e.g., LRP on networks with ReLU activation layers and gradient input).

Regarding interpretability metrics, there is no consensus in the community as the field is not mature yet. General advice would be to use different metrics and confront them to human observers, taking, for example, the methodology described in [1].
# 5 Conclusion

Interpretability of machine learning models is an important topic, in particular in the medical field. First, this is a natural need expressed by clinicians who are potential users of medical decision support systems. Moreover, it has been shown in many occasions that models with high performance can actually be using irrelevant features. This is dangerous because it means that they are exploiting biases in the training data sets and thus may dramatically fail when applied to new data sets or deployed in clinical routine.

Interpretability is a very active field of research and many approaches have been proposed. They have been extensively applied in neuroimaging and very often allowed highlighting clinically relevant regions of the brain that were used by the model. However, comparative benchmarks are not entirely conclusive, and it is currently not clear which approach is the most adapted for a given aim. In other words, it is very important to keep in mind that the field of interpretability is not yet mature. It is not yet clear which are the best methods or even if the most widely used approaches will still be considered a standard in the near future.

That being said, we still strongly recommend that a classification or regression model be studied with at least one interpretability method. Indeed, evaluating the performance of the model is not sufficient in itself, and the additional use of an interpretation method may allow detecting biases and models that perform well but for bad reasons and thus would not generalize to other settings.

# Acknowledgements

The research leading to these results has received funding from the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and reference ANR-10-IAIHU-06 (Agence Nationale de la Recherche-10-IA Institut Hospitalo-Universitaire-6).

# Appendices

## A Short Reminder on Network Training Procedure

During the training phase, a neural network updates its weights to make a series of inputs match with their corresponding target labels:

1. *Forward pass* The network processes the input image to compute the output value.
2. *Loss computation* The difference between the true labels and the output values is computed according to a criterion (cross-entropy, mean squared error...). This difference is called the loss and should be as low as possible.

3. *Backward pass* For each learnable parameter of the network, the gradients with respect to the loss are computed.

4. *Weight update* Weights are updated according to the gradients and an optimizer rule (stochastic gradient descent, Adam, Adadelta...).

As a network is a composition of functions, the gradients of the weights of a layer *l* with respect to the loss can be easily obtained according to the values of the gradients in the following layers. This way of computing gradients layer per layer is called backpropagation.

**B Description of the**
**Main Brain Disorders**
**Mentioned in the**
**Reviewed Studies**

This appendix aims at shortly presenting the diseases considered by the studies reviewed in Subheading 3.

The majority of the studies focused on the classification of Alzheimer's disease (AD), a neurodegenerative disease of the elderly. Its pathological hallmarks are senile plaques formed by amyloid-β protein and neurofibrillary tangles that are tau protein aggregates. Both can be measured in vivo using either PET imaging or CSF biomarkers. Several other biomarkers of the disease exist. In particular, atrophy of gray and white matter measured from T1w MRI is often used, even though it is not specific to AD. There is strong and early atrophy in the hippocampi that can be linked to the memory loss, even though other clinical signs are found and other brain areas are altered. The following diagnosis statuses are often used:

* AD refers to demented patients.

* CN refers to cognitively normal participants.

* MCI refers to patients in with mild cognitive impairment (they have an objective cognitive decline, but it is not sufficient yet to cause a loss of autonomy).

* **Stable MCI** refers to MCI patients who stayed stable during a defined period (often three years).

* **Progressive MCI** refers to MCI patients who progressed to Alzheimer's disease during a defined period (often three years).

Most of the studies analyzed T1w MRI data, except [49] where the patterns of amyloid-β in the brain are studied.

Frontotemporal dementia is another neurodegenerative disease in which the neuronal loss dominates in the frontal and temporal lobes. Behavior and language are the most affected cognitive functions.
Parkinson's disease is also a neurodegenerative disease. It primarily affects dopaminergic neurons in the substantia nigra. A commonly used neuroimaging technique to detect this loss of dopaminergic neurons is the SPECT, as it uses a ligand that binds to dopamine transporters. Patients are affected by different symptoms linked to motor faculties such as tremor, slowed movements, and gait disorder but also sleep disorder, depression, and other symptoms.

Multiple sclerosis is a demyelinating disease with a neurodegenerative component affecting younger people (it begins between the ages of 20 and 50). It causes demyelination of the white matter in the brain (brain stem, basal ganglia, tracts near the ventricles), optic nerve, and spinal cord. This demyelination results in autonomic, visual, motor, and sensory problems.

Intracranial hemorrhage may result from a physical trauma or non-traumatic causes such as a ruptured aneurysm. Different subtypes exist depending on the location of the hemorrhage.

Autism is a spectrum of neurodevelopmental disorders affecting social interaction and communication. Diagnosis is done based on clinical signs (behavior), and the patterns that may exist in the brain are not yet reliably described as they overlap with the neurotypical population.

Some brain characteristics that may be related to brain disorders and detected in CT scans were considered in the data set CQ500:

* Midline Shift is a shift of the center of the brain past the center of the skull.

* Mass Effect is caused by the presence of an intracranial lesion (e.g., a tumor) that is compressing nearby tissues.

* Calvarial Fractures are fractures of the skull.

Finally, one study [33] learned to predict the age of cognitively normal patients. Such algorithm can help in diagnosing brain disorders as patients will have a greater brain age than their chronological age, and then it establishes that a participant is not in the normal distribution.

## References

1. Ribeiro MT, Singh S, Guestrin C (2016) Why should I trust you?: Explaining the predictions of any Classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining - KDD '16, ACM Press, San Francisco, pp 1135-1144. https://doi.org/10.1145/2939672.2939778

2. Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: 2017 IEEE international conference on computer vision (ICCV), pp 3449-3457. https://doi.org/10.1109/ICCV.2017.371

3. DeGrave AJ, Janizek JD, Lee SI (2021) AI for radiographic COVID-19 detection selects
shortcuts over signal. Nat Mach Intell 3(7): 610-619. https://doi.org/10.1038/s42256-021-00338-7

4. Lipton ZC (2018) The mythos of model interpretability. Commun ACM 61(10):36-43. https://doi.org/10.1145/3233231

5. Xie N, Ras G, van Gerven M, Doran D (2020) Explainable deep learning: a field guide for the uninitiated. arXiv:200414545 [cs, stat] 2004.14545

6. Adebayo J, Gilmer J, Muelly M, Goodfellow I, Hardt M, Kim B (2018) Sanity checks for saliency maps. In: Advances in Neural Information Processing Systems, pp 9505-9515

7. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in Neural information processing systems, vol 25. Curran Associates, pp 1097-1105

8. Voss C, Cammarata N, Goh G, Petrov M, Schubert L, Egan B, Lim SK, Olah C (2021) Visualizing weights. Distill 6(2):e00024.007. https://doi.org/10.23915/distill.00024.007

9. Olah C, Mordvintsev A, Schubert L (2017) Feature visualization. Distill 2(11):e7. https://doi.org/10.23915/distill.00007

10. Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv:13126034 [cs] 1312.6034

11. Shrikumar A, Greenside P, Shcherbina A, Kun-daje A (2017) Not just a black box: learning important features through propagating activation differences. arXiv:160501713 [cs] 1605.01713

12. Springenberg JT, Dosovitskiy A, Brox T, Ried-miller M (2014) Striving for Simplicity: the all convolutional net. arXiv:14126806 [cs] 1412.6806

13. Zhou B, Khosla A, Lapedriza A, Oliva A, Tor-alba A (2015) Learning deep features for discriminative localization. arXiv:151204150 [cs] 1512.04150

14. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In: 2017 IEEE international conference on computer vision (ICCV), pp 618-626. https://doi.org/10.1109/ICCV.2017.74

15. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS

One 10(7):e0130140. https://doi.org/10.1371/journal.pone.0130140

16. Samek W, Binder A, Montavon G, Lapuschkin S, Müller KR (2017) Evaluating the visualization of what a deep neural network has learned. IEEE Trans Neural Netw Learn Syst 28(11):2660-2673. https://doi.org/10.1109/TNNLS.2016.2599820

17. Montavon G, Lapuschkin S, Binder A, Samek W, Müller KR (2017) Explaining non-linear classification decisions with deep Taylor decomposition. Pattern Recogn 65:211-222. https://doi.org/10.1016/j.patcog.2016.11.008

18. Montavon G, Samek W, Müller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Signal Process 73:1-15. https://doi.org/10.1016/j.dsp.2017.10.011

19. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision - ECCV 2014. Lecture notes in computer science. Springer, Berlin, pp 818-833

20. Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In: Proceedings of the 31st international conference on neural information processing systems, NIPS'17. Curran Associates, Red Hook, pp 4768-4777

21. Frosst N, Hinton G (2017) Distilling a Neural network into a soft decision tree. arXiv:171109784 [cs, stat] 1711.09784

22. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhutdinov R, Zemel R, Bengio Y (2016) Show, attend and tell: neural image caption generation with visual attention. arXiv:150203044 [cs] 1502.03044

23. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Honolulu, pp 6450-6458. https://doi.org/10.1109/CVPR.2017.683

24. Ba J, Mnih V, Kavukcuoglu K (2015) Multiple object recognition with visual attention. arXiv:14127755 [cs] 1412.7755

25. Yeh CK, Hsieh CY, Suggala A, Inouye DI, Ravikumar PK (2019) On the (In)fidelity and sensitivity of explanations. In: Wallach H, Larochelle H, Beygelzimer A, d\textquotesingle Alché-Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems, vol 32. Curran Associates, pp 10967-10978
26. Cecotti H, Gräser A (2011) Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Trans on Pattern Anal Mach Intell 33(3): 433-445. https://doi.org/10.1109/TPAMI.2010.125

27. Oh K, Chung YC, Kim KW, Kim WS, Oh IS (2019) Classification and visualization of Alzheimer's disease using volumetric convolutional neural network and transfer learning. Sci Rep 9(1):1-16. https://doi.org/10.1038/s41598-019-54548-6

28. Abrol A, Bhattarai M, Fedorov A, Du Y, Plis S, Calhoun V (2020) Deep residual learning for neuroimaging: an application to predict progression to Alzheimer's disease. J Neurosci Methods 339:108701. https://doi.org/10.1016/j.jneumeth.2020.108701

29. Biffi C, Cerrolaza J, Tarroni G, Bai W, De Marvao A, Oktay O, Ledig C, Le Folgoc L, Kamnitsas K, Doumou G, Duan J, Prasad S, Cook S, O'Regan D, Rueckert D (2020) Explainable anatomical shape analysis through deep Hierarchical generative models. IEEE Trans Med Imaging 39(6):2088-2099. https://doi.org/10.1109/TMI.2020.2964499

30. Martinez-Murcia FJ, Ortiz A, Gorriz JM, Ramirez J, Castillo-Barnes D (2020) Studying the manifold structure of Alzheimer's disease: a deep learning approach using convolutional autoencoders. IEEE J Biomed Health Inf 24(1):17-26. https://doi.org/10.1109/JBHI.2019.2914970

31. Leming M, Górriz JM, Suckling J (2020) Ensemble deep learning on large, mixed-site fMRI aatasets in autism and other tasks. Int J Neural Syst 2050012. https://doi.org/10.1142/S0129065720500124, 2002.07874

32. Bae J, Stocks J, Heywood A, Jung Y, Jenkins L, Katsaggelos A, Popuri K, Beg MF, Wang L (2019) Transfer learning for predicting conversion from mild cognitive impairment to dementia of Alzheimer's type based on 3D-convolutional neural network. bioRxiv. https://doi.org/10.1101/2019.12.20.884932

33. Ball G, Kelly CE, Beare R, Seal ML (2021) Individual variation underlying brain age estimates in typical development. Neuroimage 235:118036. https://doi.org/10.1016/j.neuroimage.2021.118036

34. Böhle M, Eitel F, Weygandt M, Ritter K, on botADNI (2019) Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer's disease

classification. Front Aging Neurosci 10(JUL). https://doi.org/10.3389/fnagi.2019.00194

35. Burduja M, Ionescu RT, Verga N (2020) Accurate and efficient intracranial hemorrhage detection and subtype classification in 3D CT scans with convolutional and long short-term memory neural networks. Sensors 20(19): 5611. https://doi.org/10.3390/s20195611

36. Dyrba M, Pallath AH, Marzban EN (2020) Comparison of CNN visualization methods to aid model interpretability for detecting Alzheimer's disease. In: Tolxdorff T, Deserno TM, Handels H, Maier A, Maier-Hein KH, Palm C (eds) Bildverarbeitung für die Medizin 2020, Springer Fachmedien, Wiesbaden, Informatik aktuell, pp 307-312. https://doi.org/10.1007/978-3-658-29267-6_68

37. Eitel F, Ritter K (2019) Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classification. In: Interpretability of machine intelligence in medical image computing and multimodal learning for clinical decision support. Lecture notes in computer science. Springer, Cham, pp 3-11. https://doi.org/10.1007/978-3-030-33850-3_1

38. Eitel F, Soehler E, Bellmann-Strobl J, Brandt AU, Ruprecht K, Giess RM, Kuchling J, Asseyer S, Weygandt M, Haynes JD, Scheel M, Paul F, Ritter K (2019) Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagation. Neuroimage: Clinical 24:102003. https://doi.org/10.1016/j.nicl.2019.102003

39. Fu G, Li J, Wang R, Ma Y, Chen Y (2021) Attention-based full slice brain CT image diagnosis with explanations. Neurocomputing 452: 263-274. https://doi.org/10.1016/j.neucom.2021.04.044

40. Gutiérrez-Becker B, Wachinger C (2018) Deep multi-structural shape analysis: application to neuroanatomy. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), LNCS, vol 11072, pp 523-531. https://doi.org/10.1007/978-3-030-00931-1_60

41. Hu J, Qing Z, Liu R, Zhang X, Lv P, Wang M, Wang Y, He K, Gao Y, Zhang B (2021) Deep learning-based classification and voxel-based visualization of frontotemporal dementia and Alzheimer's disease. Front Neurosci 14. https://doi.org/10.3389/fnins.2020.626154

42. Jin D, Zhou B, Han Y, Ren J, Han T, Liu B, Lu J, Song C, Wang P, Wang D, Xu J, Yang Z, Yao H, Yu C, Zhao K, Wintermark M, Zuo N,
Zhang X, Zhou Y, Zhang X, Jiang T, Wang Q, Liu Y (2020) Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for Alzheimer's disease. Adv Sci 7(14):2000675. https://doi.org/10.1002/advs.202000675

43. Lee E, Choi JS, Kim M, Suk HI (2019) Alzheimer's disease neuroimaging initiative toward an interpretable Alzheimer's disease diagnostic model with regional abnormality representation via deep learning. NeuroImage 202: 116113. https://doi.org/10.1016/j.neuroimage.2019.116113

44. Magesh PR, Myloth RD, Tom RJ (2020) An explainable machine learning model for early detection of Parkinson's disease using LIME on DaTSCAN imagery. Comput Biol Med 126:104041. https://doi.org/10.1016/j.compbiomed.2020.104041

45. Nigri E, Ziviani N, Cappabianco F, Antunes A, Veloso A (2020) Explainable deep CNNs for MRI-bBased diagnosis of Alzheimer's disease. In: Proceedings of the international joint conference on neural networks. https://doi.org/10.1109/IJCNN48605.2020.9206837

46. Qiu S, Joshi PS, Miller MI, Xue C, Zhou X, Karjadi C, Chang GH, Joshi AS, Dwyer B, Zhu S, Kaku M, Zhou Y, Alderazi YJ, Swaminathan A, Kedar S, Saint-Hilaire MH, Auerbach SH, Yuan J, Sartor EA, Au R, Kolachalama VB (2020) Development and validation of an interpretable deep learning framework for Alzheimer's disease classification. Brain: J Neurol 143(6):1920-1933. https://doi.org/10.1093/brain/awaa137

47. Ravi D, Blumberg SB, Ingala S, Barkhof F, Alexander DC, Oxtoby NP (2022) Degenerative adversarial neuroimage nets for brain scan simulations: application in ageing and dementia. Med Image Anal 75:102257. https://doi.org/10.1016/j.media.2021.102257

48. Rieke J, Eitel F, Weygandt M, Haynes JD, Ritter K (2018) Visualizing convolutional networks for MRI-based diagnosis of Alzheimer's disease. In: Understanding and interpreting machine learning in medical image computing applications. Lecture notes in computer science. Springer, Cham, pp 24–31. https://doi.org/10.1007/978-3-030-02628-8_3

49. Tang Z, Chuang KV, DeCarli C, Jin LW, Beckett L, Keiser MJ, Dugger BN (2019) Interpretable classification of Alzheimer's disease pathologies with a convolutional neural network pipeline. Nat Commun 10(1):1–14. https://doi.org/10.1038/s41467-019-10212-1

50. Wood D, Cole J, Booth T (2019) NEURODRAM: a 3D recurrent visual attention model for interpretable neuroimaging classification. arXiv:191004721 [cs, stat] 1910.04721

51. Thibeau-Sutre E, Colliot O, Dormont D, Burgos N (2020) Visualization approach to assess the robustness of neural networks for medical image classification. In: Medical imaging 2020: image processing, international society for optics and photonics, vol 11313, p 113131J. https://doi.org/10.1117/12.2548952

52. Tomsett R, Harborne D, Chakraborty S, Gurram P, Preece A (2020) Sanity checks for saliency metrics. Proc AAAI Conf Artif Intell 34(04):6021–6029. https://doi.org/10.1609/aaai.v34i04.6064

53. Lapuschkin S, Binder A, Montavon G, Muller KR, Samek W (2016) Analyzing classifiers: fisher vectors and deep neural networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, pp 2912–2920. https://doi.org/10.1109/CVPR.2016.318

54. Ozbulak U (2019) Pytorch cnn visualizations. https://github.com/utkuozbulak/pytorch-cnn-visualizations

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third-party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
**Chapter 23**

# A Regulatory Science Perspective on Performance Assessment of Machine Learning Algorithms in Imaging

Weijie Chen, Daniel Krainak, Berkman Sahiner, and Nicholas Petrick

## Abstract

This chapter presents a regulatory science perspective on the assessment of machine learning algorithms in diagnostic imaging applications. Most of the topics are generally applicable to many medical imaging applications, while brain disease-specific examples are provided when possible. The chapter begins with an overview of US FDA's regulatory framework followed by assessment methodologies related to ML devices in medical imaging. Rationale, methods, and issues are discussed for the study design and data collection, the algorithm documentation, and the reference standard. Finally, study design and statistical analysis methods are overviewed for the assessment of standalone performance of ML algorithms as well as their impact on clinicians (i.e., reader studies). We believe that assessment methodologies and regulatory science play a critical role in fully realizing the great potential of ML in medical imaging, in facilitating ML device innovation, and in accelerating the translation of these technologies from bench to bedside to the benefit of patients.

**Key words** Machine learning, Performance assessment, Standalone performance, Reader study, Statistical analysis plan, Regulatory science

## 1 Introduction

Machine learning (ML) technologies are being developed at an ever-increasing pace in a variety of medical imaging applications [1]. Particularly in brain imaging, the past decade has witnessed a spectacular growth of ML development for the diagnosis, prognosis, and treatment of brain disorders [2]. One of the ultimate goals of these developments is to translate safe and effective technologies to the clinic to benefit patients. Regulatory oversight plays a key role in this translation. The mission of the Center for Devices and Radiological Health (CDRH) at the US Food and Drug Administration (US FDA) is to “assure that patients and providers have timely and continued access to safe, effective, and high-quality
medical devices.”<fnref n="1" /> This chapter discusses performance assessment of machine learning algorithms in imaging applications from a regulatory science perspective. Regulatory science is the science of developing new tools, standards, and approaches to assess the safety, efficacy, quality, and performance of all FDA-regulated products.<fnref n="2" />

We begin with clarifications of the scope of this chapter. First, following an overview of the US FDA's regulatory framework for medical imaging and related ML devices, the primary topics we discuss are about concepts, basic principles, and methods for performance assessment of ML algorithms in the arena of *regulatory science* but not *regulatory policy*. As such, these topics are not necessarily relevant to every regulatory submission. The question of which components should be included in a specific regulatory submission is a regulatory decision depending on factors such as the risk of the device, impact on clinical practice, complexity of the technology, precedents, and so on and is beyond the scope of this chapter. Second, the topics are *selected* based on our experience and expertise but are not intended to be comprehensive. For example, software engineering and cybersecurity are important aspects of ML devices but are beyond the scope of this chapter. Third, as discussed in earlier chapters of this book, ML algorithms are developed for both *imaging* and *non-imaging* modalities for treating brain disorders. We focus on imaging applications. Moreover, while this book is on brain disorders, most of the discussions in this chapter are applicable to ML algorithms in general imaging applications unless noted otherwise. Lastly, while the assessment methods are well established to the best of our knowledge at the time of writing, we acknowledge that ML techniques and assessment methodologies are active areas of research and better methods may become available and adopted by researchers, developers, and regulatory agencies alike in the future. To give the readers a more specific sense of the scope of applications that are relevant to our discussions, we reviewed, via the American College of Radiology (ACR) and FDA public databases, some ML devices for brain disorders that were authorized by the FDA in recent years and summarized major scope characteristics including the imaging modalities, functionalities, and types of ML algorithms (*see* Table 1).

The rest of the chapter begins with an overview of US FDA's regulatory framework followed by topics on assessment methodologies related to ML devices in medical imaging. Rationale, methods, and issues are discussed for study design and data collection

<footnote n="1">¹ https://www.fda.gov/about-fda/center-devices-and-radiological-health/cdrh-mission-vision-and-shared-values</footnote>

<footnote n="2">² https://www.fda.gov/science-research/science-and-research-special-topics/advancing-regulatory-science</footnote>
<table><caption>**Table 1**
**Summary characteristics of exemplar FDA-cleared ML devices for brain disorders**</caption><tr><td>Modality</td><td>CT (contrast or non-contrast), CTA, MRI, PET, SPECT</td></tr><tr><td>Functionality</td><td>Triage and notification (e.g., for intracranial hemorrhage); segmentation, quantification, and feature measurements; analysis and visualization; computer-aided diagnosis; denoising, enhancement; auto-contouring/segmentation of organs at risk or tumors for radiation therapy of head and neck tumors</td></tr><tr><td>ML algorithms</td><td>Hand-crafted feature extraction and computerized classifiers; deep learning neural networks</td></tr></table>

CT computed tomography, CTA computed tomography angiography, MRI magnetic resonance imaging, PET positron emission tomography, SPECT single photon emission computed tomography. Summary based on a sampled review of public databases at ACR (https://models.acrdsi.org/) and FDA (https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices) websites. The table aims to give a general overview of the scope of devices available. For specific devices that work for a certain imaging modality with certain functionalities, please refer to the cited databases

(Subheading 3), algorithm documentation (Subheading 4), and reference standard (Subheading 5). Finally, performance assessment methodologies are overviewed including the standalone performance assessment of ML algorithms (Subheading 6), assessment of ML algorithms in the hands of clinicians (i.e., reader studies; Subheading 7), and general considerations for the statistical analysis (Subheading 8). The relationships among these topics are illustrated in Fig. 1. Performance assessment of ML devices is necessary in both premarket and postmarket environments. Premarket studies are for the assessment of safety and effectiveness before the device is authorized for marketing by a regulatory body. Some premarket studies are used in the context of device development to refine and iterate on device design. Other premarket studies are intended for review by regulatory bodies to help assess the safety and effectiveness prior to marketing authorization. Postmarket studies are for clinical use and epidemiology, maintenance, and modifications. The selected topics to be discussed in this chapter belong to premarket performance assessment.

## 2 Regulatory Framework

CDRH Learn<fnref n="3" /> provides readers an excellent resource to better understand overall medical device regulation.

**2.1 Overview**
The US FDA classifies medical devices into three classes, Classes I, II, and III. The classification determines the extent of regulatory controls necessary to provide reasonable assurance of the safety and

<footnote n="3">³ https://www.fda.gov/training-and-continuing-education/cdrh-learn</footnote>
<figure><img src="image_261.png" /><figcaption>**Fig. 1** ML performance assessment methods in the context of US FDA's regulatory framework</figcaption></figure>

effectiveness of the device. The device classification tends to increase with increasing degree of risk, and the appropriate types of controls applicable to the device depend on the device classification. There are three types of regulatory controls: general controls, special controls, and premarket authorization requirements. General controls include the basic provisions applicable to medical devices of the Food, Drug, and Cosmetic Act and apply to all medical devices. They include provisions that relate to adulteration; misbranding; device registration and listing; premarket notification; banned devices; notification, including repair, replacement, or refund; records and reports; restricted devices; and good manufacturing practices.<fnref n="4" /> Special controls apply to Class II devices and are published in the Code of Federal Regulations under the specific device type. Some examples of special controls include labeling, testing, design specifications, software life cycle documentation activities, and usability assessments.

The US FDA requirements for premarket submissions differ between the device classes. To receive FDA approval, sponsors of Class III devices, generally considered the highest risk devices, must demonstrate a reasonable assurance of safety and effectiveness. Sponsors of Class I and II device must demonstrate substantial equivalence between their new device and a legally marketed device through the premarket notification process (i.e., the 510 [k] Program), unless the product class is exempt from premarket

<footnote n="4">⁴ https://www.fda.gov/medical-devices/regulatory-controls/general-controls-medical-devices</footnote>
notification. Substantial equivalence is a comparative analysis that includes a comparison of the intended use, technological characteristics, and performance testing. For device classifications that include defined special controls (generally published in the Code of Federal Regulations or in an order granting a request for reclassification), the sponsor must also demonstrate that they have fulfilled all the necessary special controls as part of the premarket notification process and to avoid marketing an adulterated or misbranded device.

The De Novo classification process is a pathway to Class I or Class II classification for medical devices for which general controls or general and special controls provide a reasonable assurance of safety and effectiveness, but for which there is no legally marketed predicate device [3]. Devices of a new type that FDA has not previously classified are “automatically” or “statutorily” classified into Class III by the FD&C Act, regardless of the level of risk they pose or the ability of general and special controls to assure safety and effectiveness. Section 513(f)(2) of the FD&C Act allows manufacturers to submit a De Novo request to FDA for devices “automatically” classified into Class III by operation of Section 513(f) (1). In essence, a De Novo is a request for classification for a novel device that would otherwise be classified as a Class III device. During review of a De Novo request, the FDA evaluates whether general controls or general and special controls are adequate to provide a reasonable assurance of safety and effectiveness for the identified classification of the device.

FDA regulates products based upon the device characteristics (e.g., what is it? what does it do?) and the intended use of the device. The submission type and performance data necessary to obtain marketing authorization depends on the device classification, technological characteristics, and intended use. Understanding the technological characteristics is often a more straightforward exercise compared to the determination of the intended use of the product when attempting to determine the appropriate regulatory pathway and necessary supporting data. Intended use means the general purpose of the device or its function and encompasses the indications for use [4]. The indications for use, as defined in 21 CFR 814.20(b)(3)(i), describes the disease or condition the device will diagnose, treat, prevent, cure, or mitigate, including a description of the patient population for which the device is intended. The intended use of a device is one criterion that determines whether a device can be cleared for marketing through the 510(k) process or must be evaluated as a Class III device (premarket approval) or, if appropriate, a De Novo request. Section 513(i)(1) (E)(i) of the FD&C Act provides that the FDA’s determination of intended use of a device “shall be based upon the proposed labeling.” A device may have a variety of different indications for use and
intended uses (e.g., output a measurement for users, identify patients eligible for a particular treatment, estimate prognostic cancer risk, or predict a patient's response to therapy). The data needed to support these different intended uses and indications are different.

## 2.2 Imaging Device Regulation

A majority of medical image processing devices have been classified as Class II devices. Most of the software-only devices or software as a medical device that are intended for image processing have been classified under 21 CFR 892.2050 as picture archiving and communications systems. On April 19, 2021, FDA updated the name of the regulation 21 CFR 892.2050 to "medical image management and processing system." There are no published, mandatory specific special controls related to software-only devices classified under 21 CFR 892.2050, and therefore, the primary resource to understand the legal requirements for performance data associated with these devices is the comparative standard of substantial equivalence as described in detail in the guidance document on the 510 (k) Program [4]. In contrast, several devices more recently classified under the De Novo pathway have specific special controls that manufacturers marketing such devices must adhere to.

Devices originally classified via the De Novo pathway often include special controls defined in the CFR describing requirements for manufacturers of these devices. Devices that may implement machine learning that include software or software-only devices must adhere to the special controls defined in the specific regulations associated with the appropriate device class. The classification with the associated special controls is published with a Federal Register notice and appears in the Electronic Code of Federal Regulations (eCFR).<fnref n="5" /> A De Novo classification, including any special control, is effective on the date the order letter is issued granting the De Novo request [3]. For the specific examples cited below, the De Novo submission (DEN number) is cited for classifications that have not been published in CFR at the time of writing, and the associated order with special controls may be found by searching FDA's De Novo database.<fnref n="6" /> Examples include:

* 21 CFR 870.2785 (DEN200019): Software for optical camera-based measurement of pulse rate, heart rate, breathing rate, and/or respiratory rate

* 21 CFR 870.2790 (DEN200038): Hardware and software for optical camera-based measurement of pulse rate, heart rate, breathing rate, and/or respiratory rate

<footnote n="5">⁵ https://www.ecfr.gov/cgi-bin/ECFR?page=browse</footnote>

<footnote n="6">⁶ https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm</footnote>
* 21 CFR 876.1520 (DEN200055): Gastrointestinal lesion software detection system

* 21 CFR 892.2060 (DEN170022): Radiological computer-assisted diagnostic software for lesions suspicious of cancer

* 21 CFR 892.2070: Medical image analyzer

* 21 CFR 892.2080 (DEN170073): Radiological computer-aided triage and notification software

* 21 CFR 892.2090 (DEN180005): Radiological computer-assisted detection and diagnosis software

* 21 CFR 892.2100 (DEN190040): Radiological acquisition and/or optimization guidance system

The special controls associated with these regulations are intended to mitigate the risks to health associated with these types of devices. As such, many of the special controls included in these classifications relate directly to elements associated with machine learning-based software devices intended for use in diagnostics. For example, several of the regulations include special controls related to the description of the image analysis algorithm (e.g., 21 CFR 892.2060(b)(1)(i), 21 CFR 870.2785(1), 21 CFR 876.1520(5)). Many others specify elements of the performance testing and characterization. Often included in these regulations (e.g., 21 CFR 892.2060, 21 CFR 892.2070) are special controls that indicate performance must demonstrate that the device provides improved performance on a particular diagnostic task (e.g., detection, diagnosis). For new devices, these requirements generally mean FDA will require both standalone testing characterizing device performance and clinical testing demonstrating diagnostic improvement in the intended use population. For devices implementing machine learning algorithms to estimate other physiologic characteristics, standalone and clinical testing may also be required (e.g., 21 CFR 870.2785). In addition, these regulations may include special controls related to describing the expected performance of the device. Requirements associated with communicating expected device performance in labeling help to (a) mitigate the risks associated with the device and (b) communicate expectations for performance for similar devices to future device developers.

CDRH is statutorily mandated to consider the least burdensome approach to regulatory requirements or decisions. Alternative methods, data sources, real-world evidence, nonclinical data, and other means to meet regulatory requirements may be considered and accepted, when appropriate. FDA encourages innovative approaches to device design as well as mechanisms to address regulatory requirements, when appropriate. FDA takes a benefit-
risk approach to novel devices [5] and to devices with different technological characteristics [6].

CDRH provides opportunities for developers to request feedback and meet with FDA staff to obtain FDA feedback prior to an intended premarket submission [7]. These interactions tend to focus on a particular device and questions relevant to a planned future regulatory submission and may include questions about testing protocols, proposed labeling, regulatory pathways, and design and performance of clinical studies and acceptance criteria.

Device developers need to be aware of all regulatory requirements throughout a product's life cycle including investigational device requirements (e.g., 21 CFR 812), premarket requirements, postmarket requirements (e.g., 21 CFR 820), and surveillance requirements. While this chapter focuses on the premarket and performance assessment of devices, we remind the reader that regulatory requirements throughout the device life cycle should be considered.

# 3 Study Design and Data Collection

This section aims at summarizing general considerations for study design and data collection for the assessment of ML algorithms in imaging. The specific topics we focus on in this section include study objectives, pilot and pivotal studies, and issues related to data collection, including dataset mismatch and bias. Other study design considerations, such as selection of a reference standard, selection of a performance metric, and data analysis plans, are discussed in later sections.

## 3.1 Study Objectives

The first consideration in study design is the objective of the study. A general principle is that the study design should aim at generating data to support what the ML algorithm claims to accomplish. The required data are closely related to the intended use of the device, including the target patient population. Important considerations include the significance of information provided by an ML algorithm to a healthcare decision, the state of the healthcare situation or condition that the algorithm addresses, and how the ML algorithm is intended to be integrated into the current standard of care. Examples of study objectives for ML algorithms include standalone performance characterization, standalone performance comparison with another algorithm or device, performance characterization of human users when equipped with the algorithm, and performance comparison of human users with and without the algorithm.
## 3.2 Pilot and Pivotal Studies

For the purpose of this chapter, a pivotal study is defined as a definitive study in which evidence is gathered to support the safety and effectiveness evaluation of a medical device for its intended use. A pivotal study is the key formal performance assessment of ML devices in medical imaging, and the design of a pivotal study is often the culmination of a significant amount of previous work. An often overlooked, important step toward the design of a pivotal study is a pilot (or exploratory) study. Pilot studies may include different phases, including those that demonstrate the engineering proof of concept, those that lead to a better understanding of the mechanisms involved, those that may lead to iterative improvements in performance, and those that yield essential information for designing a pivotal study. When a pilot study involves patients, sample size is typically small, and data are often conveniently acquired rather than representative of an intended population [8]. Such pilot studies provide information about the estimates of the effect size and variance components that are critical for estimating the sample size for a pivotal study. In addition, a pilot study can uncover basic issues in data collection, including issues about missing or incomplete data and poor imaging protocols. For pivotal studies that include clinicians (typically radiologists or pathologists who interpret images when equipped with the ML algorithm), a pilot study can reveal poor reading protocols and poor reader training [8]. Running one or more pilot studies is therefore highly advisable prior to the design of a pivotal study.

## 3.3 Data Collection

An important prerequisite for a study that supports the claims of an ML algorithm is that the data collection process should allow the replication of the conclusions drawn from this particular study by independent studies in the future. In this regard, the composition and independence of training and test datasets and dataset representativeness are central issues.

### 3.3.1 Training and Test Datasets

Training data are defined as the set of patient-related attributes (raw data, images, and other associated information) used for inferring a function between these attributes and the desired output for the ML algorithm. During training, investigators may explore different algorithm architectures for this function and fine-tune the parameters of a selected architecture. The algorithm designer can also partition this data into different sets for preliminary (or exploratory) performance analysis, utilizing, for example, cross-validation techniques [9]. Typically, these cross-validation results are used for further model development, model selection, and hyperparameter tuning. In other words, cross-validation is typically used as an informative step before the ML algorithm is finalized. In many machine learning texts, a subset of data left out
for certain parts of algorithm design (e.g., tuning hyperparameters) is referred to as a validation set. In this chapter, we avoid calling this dataset as a validation set and call it a tuning dataset because it contradicts with the commonly used meaning of validation as “checking the accuracy of” and the definition of validation in 21 CFR 820.3⁷ as “confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled.” Since cross-validation estimates described above are typically used to modify the trained algorithm, they do not pertain to the finalized production version of the ML algorithm.

Test data are defined as the set of patient-related attributes that are used for characterizing the performance of an ML algorithm and performing appropriate statistical tests. For imaging ML software, the performance is estimated by comparing either the output of the finalized software or the interpretation of a human observer who utilizes the software to a reference standard for each case and summarizing the results for the entire dataset using appropriate metrics.

Collecting a well-characterized and representative dataset is resource-intensive, and therefore, most datasets in medical imaging are much more limited in size, compared to, for example, datasets in natural imaging or electronic health records. A general principle for dataset size is that the training dataset should be large enough to minimize overfitting and the test dataset should be large enough to provide adequate precision in testing, including adequate study power when hypothesis testing is involved. Multiple studies have shown that as the training set is gradually increased starting from a small size, overfitting is initially decreased dramatically, with diminishing returns as the dataset size gets larger [10, 11]. The size for which adding more data provides only diminishing returns depends on the complexity of the ML system and the complexity of the data space. Estimation of the test dataset size for adequate precision and study power is a classical problem in statistics, and pilot data is extremely important for this task.

### 3.3.2 Independence

A central principle in performance assessment is that the test dataset is required to be independent of the training dataset, meaning that the data for the cases in the test set do not depend on the data for the cases in the training set. It is well-known that the violation of this principle results in optimistically biased performance estimates [12]. To avoid this bias, developers typically set aside a dedicated test data for performance estimation aimed to be independent of the training dataset. There are subtle ways in which the independence principle can be violated if the test dataset is not carefully

<footnote n="7">7 https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm?fr=820.3</footnote>
selected. We discuss two such mechanisms below. The first is related to including data from a particular patient in both the training and test datasets. The second is related to performing internal validation instead of using an external validation [13] method.

A basic mechanism that can cause a dependence between the training and test sets is the inclusion of data from one patient in both datasets. This could happen if different regions of interest, different image slices, or different objects from the same patients span both the training and test datasets. Since portions of the data from the same patient are expected to be correlated, this practice will result in a statistical dependence between the training and test datasets. A straightforward principle to be followed is to include each patient's data exclusively in the training set or exclusively in the test set.

A more subtle mechanism that can cause a dependence between the training and test datasets is the way the data are sampled or the way that one dataset is partitioned into training and test datasets. Internal validation, which involves partitioning a previously collected sample into training and test datasets randomly or in a stratified way across a given attribute, may result in a dependence between the training and test datasets. Any sampled data, even if it was designed to be collected in a random manner, may not perfectly follow the true distribution of the target population due to finite sample size effects. In addition, there may be a systematic deviation in the feature distribution of a particular sample from the true distribution due to the fact that, for example, the sample may be collected only at a particular site or using only a particular or predominant image acquisition system that does not represent the true distribution. When such a dataset is shuffled and randomly partitioned into training and test datasets, knowledge about the distribution of the training data may provide unfair information about the distribution of the test dataset that would have been impossible to know had the training and test datasets been sampled independently from the true population. A practical approach to reduce this type of dependence is to sample the training and test datasets from multiple different, independent sites, a practice known as external validation [13].

### 3.3.3 Representativeness

ML algorithms are data-driven, and the distributions of the training and test data have direct implications for algorithm performance and its measurement. Ideally, training and test sets should be large and representative enough so that the collected data provides a good approximation to the true distribution in the target population. As discussed above, well-characterized and annotated medical imaging datasets are typically limited in size. When the dataset size is a constraint, informativeness of a case to be selected for training
the ML algorithm for the task at hand is an additional consideration besides representativeness [14]. Active machine learning techniques aim to proactively select training cases that can best improve model performance, based on informativeness, representativeness, or a combination of the two [15]. Active learning techniques have been applied to train ML algorithms applied to brain imaging [16, 17].

Representativeness of the test dataset is typically desirable when an unbiased estimate of the ML algorithm performance assessment is sought for the target population. For most classification problems, representativeness within each class may be sufficient, which allows designers to enrich the test datasets with classes that have smaller prevalence in the target population. For studies that aim to compare two competing arms (e.g., clinicians' image interpretation with and without ML), enrichment methods that are based on a measurement (e.g., patient or lesion characteristics, risk factors), which trade the unbiased absolute performance results for the practical ability to compare the two competing arms with possible moderate biases, are often acceptable [8]. For example, if cases that are known to be trivial to classify (or diagnose) in both arms of a comparative study are excluded from the test dataset, this will result in a bias in the absolute performance estimates for both arms but may not result in a bias in the difference or change the ranking order of the two arms under comparison, thus allowing the use of a smaller test dataset and a less resource-intensive study design. Likewise, as discussed in Subheading 6.5, when the main goal is to compare the standalone performance of two algorithms to determine which algorithm or modification performs best, it is possible to perform the comparison on a smaller enriched dataset with a careful sampling strategy that does not result in a bias in the difference of the two performance estimates.

### 3.3.4 Dataset Mismatch

Dataset mismatch is defined as a condition where training and test data follow different distributions, which is popularly known as “dataset shift” in the ML literature [18]. We prefer using “mist match” because “shift” specifically refers to adding a constant value to each member of a dataset in probability distribution theory, which does not convey all types of mismatches that the term is intended for. Dataset mismatch can also be between test data and real-world deployment data (rather than test and training) or current real-world data vs. future real-world data (e.g., due to changes in clinical practice). There may be many potential reasons for dataset mismatch, with sample selection bias and non-stationary environments cited as the most important ones [19]. Storkey [20] grouped these mismatches into six main categories, including sample selection bias, imbalanced data, simple covariate shift, prior probability shift, domain shift, and source component shift. Dataset
mismatch may result in poor performance of the trained ML algo-
rithm. In addition, especially if caused by a non-stationary environ-
ment, dataset mismatch may mean that the performance assessment
results obtained at premarket testing may no longer be valid in the
clinical environment. A first step in mitigating the effects of dataset
mismatch is to detect it. Several methods, including those based on
distance measures [21] and dimensionality reduction followed by
statistical hypothesis testing [22], have been proposed for this
purpose. Techniques for mitigating the effect of dataset mismatch
include importance weighting [23] and utilizing stratification, cost
curves, or mixture models [24], among others.

3.4 Bias

Bias is a critical factor to consider in study design and analysis for ML assessment, and here we intend to give an overview of sources of bias in ML development and assessment. Note that the general artificial intelligence and machine learning literature currently lacks a consensus on the terminology regarding bias. We consider that performance assessment of an ML system from a finite sample can be cast as a statistical estimation problem. In statistics, a biased estimator is one that provides estimates which are systematically too high or too low [25]. Paralleling this definition, we define statistical bias as a systematic difference between the average performance estimate of an ML system tested in a specified manner and its true performance on the intended population. This systematic difference may result from flaws in any of the components of the assessment framework shown in Fig. 1: collection of patient data and the definition of a reference standard (for both algorithm design and testing stages), algorithm training, analysis methods, and algorithm deployment in the clinic.

Note that the definition of statistical bias above includes sys-
tematically different results for different subgroups. ISO/IEC
Draft International Standard 22,989 (artificial intelligence con-
cepts and terminology) defines bias as systematic difference in
treatment of certain objects, people, or groups in comparison to
others, where treatment is any kind of action, including perception,
observation, representation, prediction, or decision. As such, sta-
tistical bias may result in the type of bias defined in the ISO/IEC
Draft International Standard.

We start our discussion of bias with the effect of the dataset
representativeness, which has direct implications for ML algorithm
performance and its measurement, as described above. When the
dataset is not representative of the target population, this can lead
to selection bias. For example, if all the images in the training or test
datasets are acquired with a particular type of scanner while the
target patient population may be scanned by many types of scan-
ners, this may lead to an ML algorithm performance estimate that is
systematically different from that on the intended population or
lead to different results for different subgroups. Spectrum bias,
which can be viewed as a consequence of selection bias, describes a systematic error in performance assessment that occurs when the sample of cases studied does not include a complete spectrum of patient and disease characteristics [26]. Imperfect reference standard bias and verification bias are two types of biases that are related to the reference standard (Subheading 5); the former applies to conditions in which the reference standard procedure is not 100% accurate, and the latter applies to conditions in which only subjects verified for presence or absence of the condition of interest by the reference standard are included in the training set or test set.

Aggregation bias and model design bias are two types of biases that can occur in the algorithm training stage. *Aggregation bias* is related to the information loss which occurs in the substitution of aggregate, or macro-level, data for micro-level data. Aggregation bias can lead to a model that is not optimal for any group or a model that is fit to the dominant population [27]. In ML architecture selection and algorithm training, the designer often has options for model design that may affect the objectives of accuracy, robustness, and fairness, and these objectives may have intrinsic trade-offs. *Model design bias* refers to the design choices that may amplify performance disparities among minority and majority data subgroups [28].

In addition to biases stemming from test dataset composition and the reference standard discussed above, inappropriate selection of the performance metric in the data analysis stage may result in a bias. Many metrics used for evaluation of image analysis algorithms, such as the mean squared error (MSE) for image noise reduction, do not represent the task that the ML algorithm was designed for, e.g., the detection of low-contrast objects in a noisy image. The use of an inappropriate metric may thus result in a difference between the test and true performance, e.g., a conclusion that the algorithm is helpful for its intended use when in clinical reality it is not.

Several factors may contribute to bias after a medical ML system is introduced into the clinic. One of these is the *bias due to a temporal dataset shift* [29] that may cause a mismatch between the data distribution on which the system was developed/tested and the distribution to which the system is applied. Another type of bias, sometimes termed *deployment bias* [27], may be caused by the use of a device in a manner that was not tested as part of the performance assessment and hence does not conform with the intended use of the device, e.g., off-label use. Other types of biases during deployment are also possible because of the differences in the test and clinical environments and unanticipated issues in the integration of the ML system into clinical practice.
# 4 Algorithm Documentation

## 4.1 Why Algorithm Documentation Is Important

Machine learning (ML) algorithms have been evolving from traditional techniques with hand-crafted features and interpretable statistical learning models to the more recent deep learning-based neural network models with drastically increased complexity. Appropriate documentation of ML algorithms is critically important for reproducibility and transparency from a scientific point of view. Algorithm description with sufficient details is particularly important in a regulatory setting reviewed by regulators for the assessment of technical quality, for comparing with a legally marketed device, and for the assessment of changes of the algorithm in future versions.

Reproducibility is a well-known cornerstone of science; for scientific findings to be valid and reliable, it is fundamentally important that the experimental procedure is reproducible, whether the experiments are conducted physically or in silico. ML studies for detection, diagnosis, or other means of characterization of brain disorders or other diseases are in silico experiments involving complex algorithms and big data. As such, we adopt the definition of reproducibility from a National Academies of Sciences, Engineering, and Medicine report [30] as "obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis." It has been widely recognized that poor documentation such as incomplete data annotation or specification of data processing and analysis is a primary culprit for poor reproducibility in many biomedical studies [31]. Lack of reproducibility may result in not only inconvenience or inferior quality but sometimes a flawed model that can bring real danger to patients when such models are used to tailor treatments in drug clinical trials, as reported by Baggerly and Coombes [32] in their forensic bioinformatics study on a model of gene expression signatures to predict patient response to multidrug regimens.

Appropriate documentation of algorithm design and development is essential for the assessment of technical quality. Identification of the various sources of bias discussed in Subheading 3.4 may not be possible without appropriate algorithm documentation. Furthermore, while there is currently no principled guidance on the design of deep neural network architectures, consensus on good practices and empirical evidence do provide basis for the assessment of technical soundness of an ML algorithm. For example, the choice of loss function is closely related to the clinical task: mean squared error is appropriate for quantification tasks, cross-entropy is often used for classification tasks, and so on. Moreover, the design and optimization of algorithms involve trial-and-error and ad hoc procedures to tune parameters; as such, a developer may introduce bias even unconsciously if the use of patient data and truth labels is not properly documented.
Documentation of ML algorithms is often necessary in a regulatory setting and generally required by FDA under 21 CFR 820.30 design controls. As mentioned in Subheading 2.1, comparison of technological characteristics of a premarket device with a legally marketed predicate device is one of the essential components in determining substantial equivalence for a 510(k) submission. Moreover, the ML algorithm in an FDA-authorized device is often updated, and appropriate algorithm documentation is crucial to decide if a new version has undergone major updates that would require a re-submission to the FDA.

## 4.2 Essential Elements in Algorithm Description

Many efforts in academia have been devoted to developing checklists for ML algorithm development and reporting to enhance transparency, improve quality, and facilitate reproducibility. A report from the NeurIPS 2019 Reproducibility Program [33] provided a checklist for general machine learning research. Norgeot et al. [34] presented the minimum information about clinical artificial intelligence modeling (MI-CLAIM) checklist as a tool to improve transparency reporting of AI algorithms in medicine. The journal *Radiology* published an editorial with a checklist for artificial intelligence in medical imaging (CLAIM) [35] as a guide for authors and reviewers. Similarly, an editorial of the journal *Medical Physics* introduced a required checklist to ensure rigorous and reproducible research of AI/ML in the field of medical physics [36]. Consensus groups also published the SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) as guidelines for clinical trial protocols for interventions involving artificial intelligence [37]. Also, there are undergoing efforts on guidelines for diagnostic and predictive AI models such as the TRIPOD-ML (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Machine Learning) [38] and STARD-AI (Standards for Reporting of Diagnostic Accuracy Studies–Artificial Intelligence).

Besides the abovementioned references, the FDA has published a guidance document for premarket notification [510(k)] submissions on computer-assisted detection devices applied to radiology images and radiology device data [39]. Here we provide a list of key elements in describing ML algorithms for medical imaging applications, which we believe are essential (but not necessarily complete) for understanding and technical assessment of an ML algorithm.
* Input.

The types of data the algorithm takes as input may include images and possibly non-imaging data. For input data, essential information includes modality (e.g., CT, MRI, clinical data), compatible acquisition systems (e.g., image scanner manufacturer and model), acquisition parameter ranges (e.g., kVp range, slice thickness in CT imaging), and clinical data collection protocols (e.g., use of contrast agent, MRI sequence).

* Preprocessing.

Input images are often preprocessed such that they are in a suitable form or orientation for further processing. Preprocessing often includes data normalization, which refers to calibration or transformation of image data to that of a reference image (e.g., warping to the reference frame) or to certain numerical range, e.g., slice thickness normalization. Other examples of preprocessing include elimination of irrelevant structures such as a head holder, image size normalization, image orientation normalization, and so on. Sometimes an image quality checker is applied to exclude data with severe artifacts or insufficient quality from further processing and analyses. It is important to describe the specific techniques for normalization and image quality checking. Furthermore, it is critical to make clear how cases failing the quality check are handled clinically (e.g., re-imaging or reviewed by a physician) and account for the excluded cases in the performance assessment.

* Algorithm architecture.

Algorithm architecture is the core module of a machine learning algorithm. In traditional ML techniques, hand-crafted features that are often motivated by physician's experiences are first derived from medical images. A feature selection procedure can be applied to the initially extracted features to select the most useful features for the clinical task of interest. The selected features are then merged by a classifier into a decision variable. There are many choices of the classifier depending on the nature of the data and the purpose of the classifier: linear or quadratic discriminant analysis, k nearest neighbor (kNN) classifiers, artificial neural networks (ANNs), support vector machines, random forests, etc. As such, the algorithm description typically includes the definition of features, the feature selection methods, and the specific classification model. Moreover, it is important to document hyperparameters and the method with which these hyperparameters are determined, for example, the number of neighbors in the kNN method, the number of layers in ANNs, etc.
Recently, deep learning neural network algorithms have been widely used in medical imaging applications. The NN architecture in this type of ML algorithms is composed of a large number of layers that learn to represent data at multiple levels of abstraction and automatically learns features from raw medical image data. As such, instead of sequential hand-crafted feature extraction, selection, and classification, automatic feature engineering and classification (or other types of decision-making such as quantification) are seamlessly integrated in one deep NN architecture. If a published architecture such as AlexNet, VGGNet, Inception V3, etc. is followed exactly, a succinct description is to refer to the reference. Otherwise, the architecture is typically described using a diagram with details such as the number and type of layers, the number of nodes in each layer, the activation functions, the loss function, and so on.

Sometimes hand-crafted features are combined with CNN-based automatic features by a traditional classifier (e.g., random forest) to take advantage of both the power of deep learning in information extraction and domain-specific expertise. In this situation, architecture description includes the entire pipeline, both types of information as described in the above two paragraphs.

## Algorithm Training.

ML algorithm training is the process of designing ML algorithm architecture, optimizing the parameters, and selecting the hyperparameters. Taking the popular deep neural networks as an example, the first step in training is to design an architecture or adopt one that has been proven successful in similar applications (see previous bullet). Parameters mainly refer to network weight and bias parameters for combining node outputs in one layer as inputs to nodes of the next layer. Hyperparameters include both those related to network architecture and those related to parameter optimization strategies. Network architecture hyperparameters such as number of hidden layers and units can be pre-selected and fixed if an established architecture is adopted and/or further tuned during training. Another architecture hyperparameter that has been popularly used to avoid overtraining is dropout rate, which refers to the probability of a neuron being “dropped out” in a training step (i.e., the weights are not updated) but may be active in the next step. Hyperparameters related to parameter optimization include learning rate, momentum, number of epochs, batch size, etc.

Given a set of hyperparameters, the network parameters are optimized using training images and associated truth labels. The hyperparameters are typically tuned using a separate tuning dataset. *See* Subheading 3 for discussion of training data. Again, it is important to fully describe the training process and training data as part of the algorithm description.
* Post-processing and Output.

Given the output of the main ML algorithm, some post-processing steps may be followed, for example, to transform the output to a more interpretable form. The final outputs of an ML algorithm are those that are presented to the end user such as a radiologist or other clinicians. They can be marks on the images indicating the algorithm-determined suspicious areas, a quantitative score indicating the algorithm-estimated likelihood of disease severity, and/or a binary classification indicating if the lesion is benign or malignant, etc. The algorithm description must make clear the final algorithm outputs and how they are intended to be used clinically so that appropriate validation and testing studies can be conducted.

Finally, it should be emphasized that a great description of ML algorithms not only provides these essential elements but also, more usefully, provides rationale on the algorithmic choices. Such rationale may include established good machine learning practices, evidence from similar applications, or methodological research that helps avoid overfitting, reduce bias, and improve generalizability.

# 5 Reference Standard

Rigorously developed, well-accepted reference standards (also called the “gold standard” or “ground truth”) for training and evaluating machine learning algorithms are essential to validating and characterizing the performance of machine learning algorithms. The reference standard provides a definitive or quasi-definitive characterization of the case based on information that may not be part of the machine learning input, such as biopsy or 1-year follow-up for radiological imaging oncology applications (for an example in the regulatory setting, see<fnref n="8" />). The “truthing” procedures for the cases included in validation (especially external validation) should utilize the best reference standard as recognized by the scientific community to help ensure that the performance of the device is well-characterized. The truthing process is distinct from other aspects of evaluating ML performance as the goal is to determine the “correct” characterization of each case, not to evaluate the device and reader performance in assessing a particular case.

Brain disorders often represent unique challenges to establishing appropriate reference standards. Generally, reference standard can be based on established clinical determination (including an independent modality recognized as a gold standard), follow-up clinical examination, or follow-up medical examination other than imaging. For brain disorders, the pathophysiology may be poorly

<footnote n="8">⁸ https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN170022.pdf</footnote>
understood, the progression of some disease may be slow making it difficult to reliably observe changes over time, the clinical definition of the condition may rely heavily on subjective assessments, or definitive assessments may be delayed by years (e.g., Alzheimer's disease) with different syndromes mistaken for the condition of interest (e.g., Parkinsonian-like disorders). In other words, for some brain conditions, the current best available reference standard is based on clinical determination, and confirmation from an alternative method (for instance, histopathological confirmation) may be desirable. Furthermore, ethical and pragmatic challenges of obtaining neurological tissue samples that would allow for independent pathological assessment may limit the utility of biopsy or tissue resection in many brain disorders.

In limited instances, alternatives to independent confirmation of the case “truth,” such as interpretation by a reviewing clinician (s), may be considered. Especially in brain disorders where the diagnostic criteria may already be challenging, the importance of multiple reviewing clinicians using the best possible information, even if that requires long-term follow-up, cannot be understated. For some brain disorders such as chronic traumatic encephalopathy or Alzheimer’s disease, outstanding challenges remain as the reference standard may be best assessed by biopsy or following death (i.e., autopsy). Greater biological and physiological understanding may be needed to inform the correct diagnosis early in disorder development. Using machine learning techniques to assist in this process is tempting, but the performance will generally be limited by the correctness of the reference standard. In other words, how would we assess if the ML device is outperforming the reference standard as any disagreement may be considered incorrect based on the reference truth?

Uncertainty in the reference standard needs to be accounted for in the analysis. For some machine learning devices, reference standard by expert assessment can be considered, depending on the indications for use, intended use, benefit–risk profile, and device outputs. This is often the case of ML algorithms used in segmentation tasks. In these limited instances, the reference standard from a single clinical truther remains undesirable due to potential concerns about bias or the overall performance of the truther (that is, they are not likely to be 100% accurate, especially for challenging cases). Therefore, multiple clinical truthers are desired. Truthing processes using top experts or truthing processes that weight the clinical truthers’ “accuracy” in the construction of the reference standards may also be considered (e.g., see Warfield et al. [40]).

When the truthing process involves interpretation by a reviewing clinician, the number of truthers; their qualifications, experience, and expertise; the instructions for the truthing process; and any other information should be described and documented. In instances where multiple truthers are involved, developer must
consider in advance how the interpretations of these various readers will be incorporated into the final study design and analysis. While combining the interpretation of all truthers into a single reference truth for a particular case may be appropriate in some instances, in other cases such as when the variability between truthers is high, study designs and data analysis methods that take into consideration variability in the reference standard may be appropriate. For instance, reference standard by panel discussions may face unique challenges, especially when loud voices, biases, and group dynamics may influence the outcome. On the other hand, majority vote may lead to other biases such as in segmentation where only including voxels from the majority could lead to small areas or volumes even when compared to all of the participating truthers.

Certain practices in development of the reference standard should be avoided. Often developers look for reference standards of convenience such as a single truther observing the same input data, such as a CT image, as the machine learning algorithm inputs and providing their best judgment as the underlying “truth” of the case. Truthers should not be used as readers who read those images as part of the evaluation of device performance because that can introduce considerable bias to the study results. Public data with unclear processes for establishing the reference standard or incomplete case-level data (such as data without follow-up information, without other typically assessed test results, or incomplete demographic information) frequently raises concerns about the appropriateness of the reference standard in these instances.

The reference standard should generally be based on the best available evidence for the case as recognized by the scientific community. The goal of the reference standard is to establish the “truth” for the outcome of the case. This may present challenges to cohorts where the amount of evidence may differ between cases. Requirements for the minimal amount of information available for a particular case to establish a reasonable “truth” should be defined in advance in the premarket and postmarket setting. As with overall device classification, expectations for rigor and certainty in the reference standard may increase with the device risk associated with misclassification or misdiagnosis. In a regulatory context, often more flexibility is generally permitted in the reference standard for the training data as compared to expectations of rigor in the reference standard for the validation data. Finally, the use of synthetic data is attractive as these techniques provide some opportunities for more well-characterized reference standards in some applications. While synthetic data presents an intriguing approach to addressing some challenges related to reference standards in brain disorders, this is a fairly new topic without significant experience within the current regulatory framework.
# 6 Standalone Performance Assessment

ML standalone performance is a measure of algorithm performance independent of any human interaction with the ML tool [41]. Standalone performance is the primary assessment for autonomous ML tools that make decisions without clinicians’ interactions but may be only one element of assessment for an ML algorithm used as an aid, in which case a clinical assessment of reader performance utilizing the ML may also be required, as discussed in Subheading 7. Standalone testing is also used heavily during algorithm development to benchmark performance and compare potential algorithm modifications before a “final” version is determined. This is because it is often straightforward to integrate iterative testing within the development framework. Standalone testing spans a wide range of possible implementations from initial validation of modifications using a small dataset through large-scale evaluations across multiple independent sites [42] which provides a higher level of confidence in algorithm performance.

Sometimes researchers assume that standalone testing is not important, or at least not as important, as a clinical evaluation, especially for ML-assist devices. However, standalone testing is critical even when a clinical reader study is performed because it is often conducted on larger and more diverse datasets allowing for more refined subgroup analyses and understanding of performance characteristics. It is also critical for assessing the robustness of an ML algorithm and for comparing performance across different algorithms.

In the following, we describe study design, study endpoints, and approaches for assessing standalone performance for specific types of ML tools.

## 6.1 Segmentation Assessment

Accurate segmentation of brain structures is routinely used in many neurological diseases and conditions when imaging with modalities such as CT, MRI, and PET. As an example, quantitative analysis of brain MRI has been used in assessing brain disorders such as Alzheimer's disease, epilepsy, schizophrenia, multiple sclerosis (MS), cancer, and infectious and degenerative diseases [43]. Often brain assessment quantifying change over time requires the segmentation of brain tissue or anatomy. We define segmentation as the process of partitioning a brain image or image volume into multiple objects defined by a set of voxels unique to each structure or object of interest.

There have been various methods proposed for assessing how well an ML algorithm characterizes objects and how one segmentation algorithm compares to another. Zhang discusses three basic approaches to assessing segmentation algorithms in general [44]. This includes analytical methods, goodness methods, and
discrepancy methods [45]. Analytical methods consider the principles, requirements, utilities, and complexity of segmentation algorithms but can be quite difficult to apply, especially to DL-based segmentation because not all algorithm properties are easily obtained. Goodness methods evaluate segmentation performance by judging the segmented images based on certain quality measures established according to human intuition and include measures such as inter-region uniformity, inter-region contrast, and region shape [45]. Discrepancy methods quantify the difference between segmented objects and a reference standard segmentation. They are the most common type for assessing segmentation algorithms with the caveat that often a ground-truth segmentation is not available. In this case, algorithm segmentations are then compared to human segmentations where the human segmentation is considered the reference standard. Since human segmentations of brain anatomy and structure can be quite variable, segmentations by multiple truthing readers are often collected, and an aggregated reference [40, 46] is used, or the agreement or interchangeability of the algorithm with a truthing reader is assessed.

The remainder of this subsection describes a few common segmentation metrics where we assume a hard segmentation and a single reference standard. A hard segmentation means a voxel is either part of the segmentation or not (this is in contrast to a soft or fuzzy segmentation which means that each voxel is assigned a probability of being part of the segmentation).

An example of a 2D segmentation S and reference segmentation R is shown in Fig. 2 for image X. The false-positive (FP), true-positive (TP), false-negative (FN), and true-negative (TN) regions are also shown. Taha and Hanbury provide a nice overview of 20 segmentation metrics used for discrepancy assessment [47]. Please refer to this paper for more details on many segmentation assessment approaches including methods for assessing fuzzy segmentation algorithms [47]. We next discuss some of the discrepancy assessment approaches frequently used in the literature.

Overlap indexes assess a segmentation by how well it overlaps with the reference. We define some basic overlap metrics below using TP, TN, FP, and FN as voxel counts in the definitions.

* Voxel true-positive rate (TPR), sensitivity, recall: proportion of correctly segmented reference voxels.

$$TPR = \frac{TP}{TP + FN}$$

* Voxel true-negative rate (TNR), specificity: proportion of correctly segmented background voxels.

$$TNR = \frac{TN}{TN + FP}$$
<figure><img src="image_262.png" /><figcaption>Fig. 2 Diagram of a segmented object (blue solid line) overlaid by the reference segmentation (red dashed line). The false-positive (FP) voxels (green hashed region), true-positive (TP) voxels (yellow hashed region), false-negative (FN) voxels (purple wave region), and true-negative (TN) voxels (gray region) are shown in the figure as well</figcaption></figure>

* Voxel accuracy [45]: proportion of correctly segmented voxels (including both reference and background voxels).

$$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$$

* Dice similarity coefficient (DSC), F1 metric [48]

$$DSC = \frac{2TP}{2TP + FP + FN} = \frac{2JI}{1 + JI}$$

* Jaccard index (JI), intersection over union (IoU) metric [49]

$$JI = \frac{TP}{TP + FP + FN} = \frac{DSC}{2 - DSC}$$

The Dice coefficient (DSC) is the most widely used performance metric for characterizing medical image volume segmentations including brain segmentations and can also be used to assess the reproducibility of multiple annotations [47]. The Jaccard index is another common assessment metric. JI and DSC are monotonically related with DSC always having a larger value than JI except at 0 and 1 when the two are equal. However, they have different
properties when averaging performance across multiple segmentations where Jaccard penalizes large segmentation errors more than Dice (somewhat similar to how an L2 norm penalizes larger error more than an L1 norm) [50].

Accuracy is another commonly reported metric, but accuracy is often dominated by a large disparity in the number of reference and background voxels within an image. Accuracy can be high even for poor overlap in the segmentation and reference when the vast majority of voxels in the image are background. This can make it difficult to differentiate between algorithms based only on accuracy differences. A similar observation can be made for specificity or, more generally, for any segmentation metrics involving TN. Indeed, since most voxels are background, TN can be very large. Finally, the definition of the background is not always straightforward and can sometimes be arbitrary (for instance, if the background depends on the field of view of the image).

Distance-based metrics are useful when the boundary of the segmentation is critical [51]. They assess the distance between the segmentation boundary and the reference boundary taking into account the spatial position of the boundary voxels [47]. Some common distance metrics include:

*   Hausdorff distance (HD) between two voxel sets $B_S$ and $B_R$ (sets of boundary voxels) [47]

$$HD = \max_{r \in B_R} \min_{s \in B_S} \|r - s\|$$

*   Mahalanobis distance (MHD) between two voxel sets $B_S$ and $B_R$.

$$MHD = \sqrt{(\mu_{B_S} - \mu_{B_R})^2 + (\mu_{B_S} - \mu_{B_R})^2}, \text{ where } \mu_{B_S} \text{ and } \mu_{B_R} \\ \text{are the means of the point sets and S is the common covariance matrix of the two sets [47]}$$

There are additional segmentation assessment metrics including volume metrics, information theoretic metrics (e.g., mutual information), probabilistic metrics (e.g., intraclass correlation coefficient [ICC]), and pair counting metrics that can also be used to assess the quality of a segmentation algorithm or for comparing multiple segmentations [47].

**6.2 Classification Assessment**

Classification ML are algorithms designed to parse brain images and data into unique categories. Often the task is differentiating two groups (e.g., cancer versus non-cancer patients), but classification can also be multiclass (e.g., differentiating astrocytoma, glioblastoma, and meningioma brain tumors). The outputs of an ML algorithm can be discrete classes (e.g., via decision tree) or a continuous or a quasi-continuous score (e.g., output of a linear classifier and many DL methods) for an image. As with all ML, the classifier output needs to be assessed and properly interpreted, so
ML performance is understood in the correct context. Tharwat [52] and Hossin and Sulaiman [53] have nice summaries of classification analysis methods. They discuss various performance metrics along with information on how and when each metric might be most effectively used in classifier assessment.

In the remainder of this subsection, we concentrate on binary classifier assessment that includes a wide range of statistical metrics for assessing classifier performance starting with operating point metrics defined directly from discrete ML outputs and moving to more complex metrics based on thresholding a continuous ML output score.

Some basic prevalence-independent metrics (i.e., metrics that do not depend on the prevalence of diseased cases in the standalone database) are described below where TP, TN, FP, and FN are case counts here.

* True-positive rate (TPR), sensitivity, recall

$$TPR = Se = \frac{TP}{TP + FN}$$

* True-negative rate (TNR), specificity

$$TNR = Sp = \frac{TN}{TN + FP}$$

Likelihood ratios are aggregate measures combining sensitivity and specificity. The positive/negative likelihood ratio is the ratio of the probability of a person who has the disease testing positive/ negative over the probability of a person who does not have the disease testing positive/negative. They are defined as:

* Positive likelihood ratio (LR⁺)

$$LR^{+} = \frac{TPR}{1 - TNR}$$

* Negative likelihood ratio (LR⁻)

$$LR^{-} = \frac{1 - TPR}{TNR}$$

Other operating point metrics depend on the prevalence of disease in the test dataset. They include:

* Positive prediction value (PPV), precision

$$PPV = Precision = \frac{TP}{FP + TP}$$

* Negative prediction value (NPV)

$$NPV = \frac{TN}{FN + TN}$$
* Accuracy

$$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$$

* F₁ score

$$F_1 = 2 \cdot \frac{PPV \times TPR}{PPV + TPR}$$

These metrics are most appropriate when assessing ML performance in a dataset representing the true clinical population because of their prevalence dependence. They must be interpreted with caution when applied to enriched datasets especially when extrapolating the estimated classification performance to the clinical environment.

For continuous ML scores where a final classification is based on applying a threshold to the output scores, there are aggregation measures that more completely characterize overall classifier performance for a binary task. A common choice is receiver operating characteristic (ROC) analysis which characterizes performance for all possible operating points of the classifier. An ROC curve plots TPR as a function of the false-positive rate (FPR = 1-TNR) when the threshold on the classifier output is varied over the complete range of possible output scores [54–56]. An example of an ROC curve is shown in Fig. 3. The advantage of the ROC curve is it shows the benefit (i.e., TPR) as a function of all possible risk values (i.e., FPR) such that a much more complete understanding of the benefit-risk trade-off at all operating points is provided [52].

To facilitate statistical comparisons and to benchmark performance, summary performance can be estimated from ROC curves with the most popular being the area under the ROC curve (AUC) and the partial AUC (PAUC area under just a portion of the ROC curve) [41]. However, the ROC curve should always be plotted to allow for a visual assessment of an individual algorithm's performance or to facilitate a comparison across algorithms. This allows the trade-off across the full range of the ROC curve to be visualized.

Parametric and nonparametric statistical methods are available to both estimate AUC/PAUC and their uncertainties. These approaches allow for statistical comparisons in performance among multiple ML algorithms. There is substantial literature on statistical method for assessing and comparing ROC performance. A great summary of approaches can be found in a report on ROC by the International Commission on Radiation Units and Measurements (ICRU) [57].
<figure><img src="image_263.png" /><figcaption>**Fig. 3** Plot of an ROC curve for an ML algorithm in a binary classification task. The ROC curve (blue line) shows the trade-off between sensitivity (TPR) and specificity (FPR) for all possible operating points. Both the AUC (includes both shaded regions) and the PAUC for sensitivity ≥0.8 are shown in the figure</figcaption></figure>

**6.3 Abnormality**
**Detection Assessment**

ML detection algorithms mark locations or regions of an image that may reveal abnormalities [41]. Examples of basic ML detection include ML-based bounding boxes or segmentations of potential brain lesions or markers indicating potential brain lesions in an MR or CT scan. Often ML detection outputs include not only localization information but also a confidence score or class determination for the identified regions such that the ML includes both detection and classification functionalities. In the remainder of this subsection, we concentrate on assessing only detection performance without addressing any other potential components of an ML algorithm's output. However, we will still use the ML detection confidence scores, when available, to expand the range of possible performance metrics available for standalone assessment.

Similar to classification metrics, there are a wide range of metrics available for assessing detection performance. Basic detection operating point metrics, usually based on thresholding a continuous ML score for each region, include counts of object-based true-positive (TP), false-positive (FP), and false-negative (FN) detections using the basic definitions in Subheading 6.2 above. Note that object-based true-negative (TN) detections are
generally not estimable in ML detection because there is an infinite (or at least an extremely larger number) of possible TN locations within an image [41]. In addition, ML detection assessment is complicated by the need for a predefined rule (method and threshold) for determining a “correct” detection based on the overlap of a bounding box/segmentation with a reference standard region or the distance from an ML marker to a reference standard object (e.g., distance to the centroid of a reference standard). The overlap metric is often based on the intersection over union (IoU) for bounding boxes/segmentations with a reference standard object and Euclidean distance for markers. However, other potential overlap metrics and criteria may be justifiable for various detection tasks.

Based on the number of TP, FP, and FN detection counts, some basic summary operating point metrics include the true-positive rate (TPR) (i.e., recall) and positive predictive value (PPV) (i.e., precision) that are defined similarly as those in Subheading 6.2 but with the unit of regions/cases instead of voxels and the number of FPs per case (or another appropriate unit of interest) since individual cases often include multiple images and abnormalities of interest [41]. For example, an MR exam of the head may include multiple MRI sequences (e.g., T1, T2) such that it is possible to report ML detection performance on a per-patient, per-view (sequence), or per-abnormality (object) basis. The unit of performance should be clearly defined and justified with per-abnormality (or object) performance typically being reported for most image-based ML detection devices especially when only a single exam is available per patient.

Analogous to classification tasks, aggregation metrics that more completely characterize overall ML detection performance are used when a confidence score is available for each detection. ROC analysis is not generally used for ML detection assessment because, as mentioned previously, TNs are not estimable. Therefore, alternate methods have been developed including the free-response receiver operating characteristic (FROC) analysis. FROC accounts for localization and detection of an arbitrary number of abnormalities within an image set [58]. FROC curves plot the fraction of correctly localized lesions as a function of the average number of FPs across the full range of confidence scores for an ML detection algorithm [59]. An example of FROC curve is shown in Fig. 4.

The plot in Fig. 4 shows a nonparametric FROC curve. Parametric FROC methods have been developed using maximum likelihood methods [60–62]. Similar to ROC analysis, FROC area-based metrics can serve as summary performance metrics, but, since the number of FPs in FROC are not bounded, the area under the curve is not limited. This complicates the use of the full area under the FROC curve as a summary figure of merit. Therefore, alternate area-based figures of merit have been developed to summarize and compare FROC performance curves.
<figure><img src="image_264.png" /><figcaption>**Fig. 4** Plot of an FROC curve for an ML detection algorithm. The FROC curve (blue line) shows the trade-off between object detection sensitivity (TPR) and the number of FPs per patient for all possible operating points. The full area under the FROC curve is not well defined, but a partial area may facilitate comparisons across algorithms. The figure shows a PAUC (shaded region) for ≤3.0 FPs/patient. However, AFROC-based summary metrics are more commonly used for characterizing/comparing FROC performance</figcaption></figure>

The area under the alternate FROC (AFROC) curve with a jackknife method (JAFROC) was developed to provide confidence interval estimates and facilitate statistical performance comparisons across algorithms [56, 61]. AFROC provides an alternative way to summarize FROC data where the fraction of negative images falsely called positive are computed based on the highest FP score for each image in the dataset [58]. In this way, the unlimited x-axis of FROC curves is now bounded at 1 as shown in Fig. 5, and the area under the curve is well defined. Chakraborty's jackknife FROC (JAFROC) metric is the area under this AFROC calculated using a jackknife approach [56, 61].

Another common aggregate assessment for ML detection performance is the precision–recall (P–R) curve (see Fig. 6) which plots the trade-off between precision and recall across the full range of ML detection algorithm confidence scores [63].

As a reminder, precision (PPV) is a measure of how well the ML detection algorithm identifies only relevant abnormalities, while recall (TPR) is a measure of how well the algorithm finds all abnormalities. A better ML detection algorithm will have a higher precision at a fixed recall. Therefore, a larger area under the P–R
<figure><img src="image_265.png" /><figcaption>**Fig. 5** Plot of an AFROC curve for the same ML detection algorithm given in Fig. 4. The AFROC curve (blue line) shows the trade-off between sensitivity (TPR) and the false patient fraction (fraction of patients with at least one FP) for all possible operating points. The area under the AFROC curve (shaded region) is often used to facilitate comparisons across object detection algorithms</figcaption></figure>

<figure><img src="image_266.png" /><figcaption>**Fig. 6** Plot of a P-R curve for the same ML object detection algorithm as in Figs. 4 and 5. The P-R curve shows the trade-off in precision (PPV) as a function of recall (TPF). The area under the P-R curve (AUCPR) is an aggregate summary metric, for characterizing and comparing P-R curves across object detection algorithms</figcaption></figure>
curve indicates improved performance compared to a competing algorithm, at least when the two P-R curves do not cross. The area under the P-R curve (AUCPR) is again an aggregate summary metric with the average precision (AP) as one estimation method developed in the information retrieval literature and has been used as a performance metric in ML Grand Challenges assessing ML localization algorithms [64, 65]. Nonparametric P-R curve and AP are commonly reported with one definition of AP given below.

* Average precision

$$ 
\begin{aligned}
AP &= \sum_{n=1}^{N} (R_{n+1} - R_{n}) P_{\text{interp}}(R_{n+1}), \text{ where } P_{\text{interp}}(R_{n+1}) \\
&= \max_{\hat{R}: \hat{R} \ge R_{n+1}} P(\hat{R})
\end{aligned}
$$

Another approach is to use an 11-point interpolation by averaging the maximum precision for a set of 11 equally spaced recall levels [0, 0.1, 0.2, ..., 1] [63]. Parametric [66] and semi-parametric [67] methods for fitting the P-R curve and methods for estimating the AUCPR (e.g., trapezoidal estimators, interpolation estimators) have also been reported in the literature.

One of the complications in assessing an ML algorithm for abnormality detection is the need for determining a “correct” detection based on either an overlap measure for a bounding box/segmentation output or a distance metric for a marker output. Since ML algorithm performance depends on the “correct” detection criterion defined by an empirically chosen overlapping or distance parameter, a sensitivity analysis of the standalone performance across a range of overlap parameters is helpful to confirm that the performance estimate is reasonably stable or to at least understand how the choice of the criterion impacts performance. Moreover, while we have concentrated on detecting a single abnormality here, the abnormality detection metric discussed above can be generalized to multiple-object detection problems by reporting overall performance or assessing performance individually for each type of abnormality and averaging across abnormality types.

## 6.4 Triage Assessment

A triage ML algorithm analyzes images for findings suggestive of a target clinical condition, but instead of making a diagnosis or detection on the image, the algorithm is limited to generating a notification in the reading worklist or communicating directly to a specialist that a patient has a potential time-sensitive condition. Triage ML devices are often called computer-assisted triage and notification (CADt) devices. CADt is designed to allow a full clinical review earlier in the workflow than without the ML notification, given a true-positive (TP) finding by the algorithm. This can
benefit patients for conditions that are time critical by providing
more timely care. For example, in cases involving suspected large
vessel occlusion (LVO) stroke, a notification from an effective
CADt device could allow a neuro-interventionalist to expeditiously
treat the clot, potentially reducing some associated morbidity and
mortality.

Another situation that CADt devices are useful is in a busy
clinical environment where a large number of cases are queued
waiting for clinician review. Instead of reading the cases in a first-
in-first-out (FIFO) fashion, the clinician can review CADt flagged
cases before non-flagged cases, thereby reducing the waiting time
of the diseased patients.

In both situations described above, the sensitivity of the CADt
for the target condition is critical so that the truly diseased patients
benefit from earlier diagnosis and treatment. However, specificity is
also important for the following reasons. An ML algorithm is
unlikely to have 100% sensitivity, i.e., there are inevitably false-
negative patients in the queue. These patients may be significantly
delayed compared with FIFO reading if the triage algorithm has a
large false-positive rate (i.e., low specificity). Moreover, too many
false alarms may lower the vigilance of a specialist which in turn may
affect their performance on the true-positive patients. Therefore,
the metric sensitivity and specificity should be used as a pair to
assess CADt performance. In the same spirit, the overall capability
of the ML algorithm in distinguishing between patients with the
condition and those without can be assessed via ROC analysis and
the area under the ROC curve.

Despite its usefulness in evaluating a CADt device, the (sensi-
tivity, specificity) pair and ROC performance are metrics of diagno-
sis and, at best, indirect measures of the true clinical effectiveness of
an ML triage, i.e., reduction of the waiting time for patients with
the target time-sensitive condition. Quantitative assessment of the
clinical effectiveness of CADt devices in accelerating the review of
patient images with the condition of concern is an open question.
Among the efforts we are aware of, Thompson et al. [68] are
developing an analytical approach based on the queueing theory
to quantify the wait-time-saving of CADt. Under a clinical work-
flow model parameterized by disease prevalence, patient arrival rate,
radiologist service rate, and number of radiologists on-site, their
method allows computation of the average waiting time saved for a
truly diseased patient due to the use of the CADt device where
CADt performance is characterized by its sensitivity and specificity
in diagnosing the condition of interest. This approach can poten-
tially be useful in assessing the clinical effectiveness of CADt algo-
rithms but requires further development and validation. Likewise,
alternate approaches for assessing true CADt effectiveness in a
clinical setting should be an area of continued research.
**6.5 Utility of Standalone Performance Assessment**

As mentioned previously, ML standalone assessment is primarily used to benchmark algorithm performance and compare with other ML algorithms or prior versions of the same algorithm to determine a performance change. Once a standalone dataset has been established and referenced, and the various performance metrics and criteria set, standalone testing can generally be applied in an efficient manner. Therefore, standalone testing is an important tool for assessing the potential bias in an ML algorithm. When a large diverse standalone dataset containing a range of patients with various demographic characteristics, a wide range of disease conditions, and the full range of acquisition technologies and protocols is available, ML performance can be estimated and compared both overall on the full dataset and in separate subgroups within this larger population to help identify where the ML may perform better and worse.

The standalone testing is also a critical tool for confirming a potential bias or disparity when this disparity is hypothesized, through specifically targeting the assessment to that subgroup of interest. Through standalone testing, ML performance can quickly be evaluated on the specific subgroup to determine if concern is warranted. The data requirements for this type of focused subgroup assessments may not need to be unusually large if the goal is to identify large disparities in performance when the ML algorithm is suspected to be performing poorly. Obviously, identifying more nuanced differences in performance across subgroups requires larger datasets.

Finally, standalone testing is a great tool for comparing ML algorithms. Again, it is ideal to obtain a large diverse real-world dataset to fully assess and benchmark an ML algorithm, but comparison can often be performed on much smaller enriched datasets where the main goal is to determine which algorithm or modification performs best, especially in the developmental phases of an algorithm's life cycle.

**6.6 Modifications and Continuous Learning**

One of the potential advantages of ML is its ability to quickly learn from new data such that it can remain current to changing patient demographics, clinical practice, and image acquisition technologies. This ability may result in large numbers of updates to an ML algorithm after it becomes available for clinical use. However, each update requires a systematic assessment. Modifications can range from infrequent algorithm updates all the way to continuously learning ML that adapts or learns from real-world experience/data on a continuous basis. This presents a challenge to both ML developers and regulatory bodies such as the FDA.

FDA's traditional paradigm of regulating ML devices is not designed for adaptive technologies, which adapt and optimize performance on a rapid timescale. With this in mind, the FDA is exploring a new, total product life cycle (TPLC) regulatory
approach that may potentially accommodate the rapid modification
cycle of ML algorithms allowing for their efficient improvement
and adaptation to the changing clinical environment while still
providing effective safeguards that meet FDA's statutory require-
ments to ensure safety and effectiveness. To this end, the FDA
released a proposed regulatory framework for modifications to
AI/ML software as a medical device (SaMD) in 2019 as a discus-
sion paper [69] requesting feedback from the public on the pro-
posed framework. The proposed TPLC approach is based on [69]:

* The assurance of quality systems and good machine learning practices (GMLP).

* An initial premarket assurance of safety and effectiveness.

* A limited set of SaMD pre-specifications.

* A well-defined algorithm change protocol.

The algorithm change protocol is defined as the specific meth-
ods that will be used to achieve and appropriately control the risks
of the SaMD pre-specifications [69].

This proposed framework is still under development, but the
FDA did provide more details on their potential approach with the
release of the AI/ML SaMD Action Plan in January 2021. The
Action Plan was developed in response to the stakeholder feedback
received on the proposed framework and to support innovative
work in the regulation of medical device software and other digital
health technologies.<fnref n="9" />

In response to the FDA's proposed framework, Feng et al. have
been working to frame an AI/ML algorithm change protocol as an
online statistical hypothesis testing problem [70]. The goal of their
work was to investigate how “biocreep” resulting from repeated
testing and adoption of modifications might lead to a gradual
deterioration in ML performance. Feng et al. were able to show
that biocreep would regularly occur when using policies with no
error-rate guarantees but policies that included error-rate control
were able to control biocreep without substantially impacting the
ability to approve beneficial modifications [70]. This was an
in-depth study of a very limited scope of potential ML modification
problems as indicated by Feng et al. [70], and there remains a great
deal of work to address the challenges around other types of mod-
ifications and conditions. The scientific community, especially inter-
disciplinary teams of clinicians, statisticians, and domain experts,
are encouraged to take on this interesting and complex ML
problem [71].

<footnote n="9">⁹ https://www.fda.gov/media/106331/download</footnote>
images, then the average of the performance estimates $\bar{A}$ in these repeated experiments would be $A$, and the population performance $A$ would be within the estimated CIs 95% of the time. In this sense, we say the performance estimate “$\bar{A}$ [$L_{\bar{A}}, U_{\bar{A}}$]” generalizes to both the reader population and the patient population, i.e., the performance estimate represents the expected performance of a random reader reading a random case using a medical device (e.g., an ML algorithm). To distinguish from a fixed-reader study, a random-reader study is often referred to as a multi-reader multi-case (MRMC) study. As a passing note, this discussion also indicates that it is critical to specify the intended patient population and user population of a device so that a study can be designed to collect data from those populations.

The statistical methodology for generalizing the performance of an imaging device to both the population of readers and the population of cases was first developed by Dorfman, Berbaum, and Metz (DBM) [72]. Since then, many methodologies have been developed for the analysis of MRMC data such as the Obuchowski and Rockette (OR) [73] model based on a correlated ANOVA model; the bootstrap method by Beiden, Wagner, and Campbell [74]; and the *U* statistic method by Gallas [75]. Relationships among these methods have also been investigated [76, 77]. These early developments of MRMC analysis methods have focused on the area under the ROC curve (AUC) as a performance metric; some of these methods (e.g., OR and *U* statistic methods) have been extended to binary performance metrics [78], and all these methods have been validated with simulation studies [79] [80]. Some of these methods also have publicly available software tools, such as the integrated and updated OR-DBM method<fnref n="10" /> and the *U* statistic method.<fnref n="11" />

The most widely used MRMC study design for comparing two modalities (e.g., without ML versus with ML) is the fully crossed (FC) design, in which every reader reads every case in both modalities. The advantage of pairing both readers and cases across two modalities is that it builds a positive correlation between the performance estimates of the two modalities, thereby reducing the variability of the performance difference and enhancing the power of detecting the performance difference. This reduction of variability can be easily appreciated by a simple formula

<footnote n="10">¹⁰ Software | Medical Image Perception Laboratory Department of Radiology (uiowa.edu): https://perception.lab.uiowa.edu/software-0</footnote>

<footnote n="11">¹¹iMRMC: Software to do multi-reader multi-case analysis of reader studies: https://github.com/DIDSR/iMRMC</footnote>
$$ \mathrm{Var}[\hat{A}_1 - \hat{A}_2] = \mathrm{Var}[\hat{A}_1] + \mathrm{Var}[\hat{A}_2] - 2\rho\sqrt{\mathrm{Var}[\hat{A}_1]\mathrm{Var}[\hat{A}_2]}, $$

where “Var” denotes variance; $\hat{A}_1$ and $\hat{A}_2$ are performance estimates for, e.g., without ML and with ML, respectively; and $\rho$ is the correlation between $\hat{A}_1$ and $\hat{A}_2$ and is positive under normal circumstances. Pairing cases from two modalities sometimes is not advised due to safety concerns, for example, if both imaging modalities involve ionizing radiation to patients, imaging the patient twice may raise dose concerns. Fortunately, this is not generally an issue for the assessment of ML algorithms, and pairing cases in a “without ML versus with ML” comparison is feasible in many diagnostic situations.

The FC design has been regarded as the most powerful design in the sense that it makes full use of available readers and cases in collection of information. However, practically the workload of a radiologist may be limited, and oftentimes an investigator may have more cases than what readers can afford to read. Moreover, as multi-site evaluation becomes popular for better generalizability, the transfer of cases among different clinical sites can be logistically demanding. To overcome these limitations, Obuchowski [81] investigated the split-plot design, where different groups of readers read different groups of cases. The combined reader/case group can still be paired across modalities to reduce the variability of performance difference. Figure 7 provides a visual illustration of the FC design and the paired split-plot (PSP) design. What might be surprising is that the PSP design can be more powerful than the FC design, as shown by Hillis et al. with empirical data [82] and Chen et al. with both theoretical analysis and real-world data [83]. This may sound like a paradox since the FC design is regarded as “the most powerful design,” but it is not. Referring to Fig. 7, suppose we have a certain number of readers and each of them can read the same number of cases. In the FC design, all the readers read the same cases (see Fig. 7, top), whereas, in the PSP design, readers are partitioned into two groups with each group reading the same number of cases from two different case sets (see Fig. 7, bottom). As such, the two designs involve the same amount of workload (i.e., number of image interpretations). However, the PSP design has reduced variability in performance estimates and performance difference estimates and hence increased statistical power, as proved by Chen et al. [83] because of the inclusion of additional cases. One way to understand this is that, with the same workload, reading difference cases (by half of the readers) gains more information than reading the same cases. This is also consistent with a common statistical sense: when we have more cases, the variability of the “mean” measured on the cases is reduced. In summary, the FC design is the most powerful given the same
Fully Crossed Design

<figure><img src="image_267.png" /></figure>

Paired Split-Plot Design

<figure><img src="image_268.png" /><figcaption>**Fig. 7** Illustration of the fully crossed design and the paired split-plot design. The squares with grid can be understood as the data matrix collected in the reader study with each row representing a case, each column representing a reader, and each data element representing the rating of the case by the reader</figcaption></figure>

number of readers and cases, but the PSP design can be more powerful *given the same number of image interpretations* with a price of collecting extra cases [83].

The design of an MRMC reader study involves a great deal of considerations including patient data collection (*see* Subheading 3), establishment of a reference standard (*see* Subheading 5), and many other aspects such as the recruitment and training of readers and
reading session design, e.g., sequential reading, where the readers
read images with ML turned on immediately after reading without
ML, or concurrent reading, where readers read images with ML
turned on from the very beginning and this is typically compared
against readers' performance reading images without ML in a
separate session. It is worth noting that the discussion of the
performance testing here is generally based on ML systems that
are intended to "aid" or interface with an expert radiologist. The
intended use of a model may warrant additional testing considera-
tions related to human factors and human interpretability depend-
ing on how the model is integrated into the clinical workflow.
Moreover, MRMC studies for the assessment of ML in imaging
are often retrospective and controlled "laboratory" studies, in
which typically only information related to the device of interest is
presented to the readers (e.g., "image only" versus "image plus ML
output"), whereas in real-world clinical practice, more information
is often available to the physician, e.g., patient history, clinical tests,
and/or other types of imaging exams. The diseased cases are often
enriched when the natural prevalence is low in controlled labora-
tory studies. The purpose of such designs is to remove certain
confounders and increase the statistical power to study the impact
of the ML algorithm itself rather than the "absolute" performance
of clinicians in the real world (as discussed in Subheading 3).
However, consideration should still be given to ensure the study
execution is as close to the clinical environment as possible and
identify/mitigate potential biases, for example, the readers should
be trained to use the ML algorithm as if they were instructed in the
clinic. It is also important to randomize cases, readers, and reading
sessions to minimize bias. For more details on the design of MRMC
studies, interested readers can refer to an FDA guidance document
[84], a consensus paper by Gallas et al. [8], as well as a tutorial
paper by Wagner et al. [55].

8 Statistical Analysis

The statistical analysis plays a critical role in the assessment of ML
performance but may be under-appreciated by many ML develo-
pers. For example, there are still publications that present point
estimates of ML performance without quantification of uncertain-
ties (standard deviations, confidence intervals). Even if uncertainty
estimates are provided, the methods of uncertainty estimation are
sometimes unclear or even inappropriate. Another example is the
re-use of test data. One may follow the good practice of using
independent datasets for ML training and testing. However, if the
test data is repeatedly used, the seemingly innocent good practice
may introduce optimistic bias to the performance estimate or even
lead to a spurious discovery because the repeatedly measured
performance on the test dataset may inform training of the algo-
rithm to adaptively fit the test data [85, 86]. As the quote goes, “if
you torture the data long enough, it will confess.” The lesson is,
without following appropriate statistical principles, ML developers
may be led to a blind alley due to statistical pitfalls: comparisons are
made without statistical rigor, conclusions are drawn without
appropriate data to substantiate, and spurious findings out of over-
fitting are celebrated. Statistical practices have a major impact on
the ability to conduct reproducible research.

A good practice to avoid such pitfalls is, for any performance
assessment study—either standalone performance assessment or an
MRMC study—to *pre-specify* a statistical analysis plan (SAP) with
valid statistical methods. The word “pre-specify” is emphasized
because post hoc analyses can inflate the experiment-wise type I
error rate and endanger the scientific validity of an otherwise well-
designed and well-conducted study. Below we list exemplar ele-
ments in an SAP for ML development and assessment. We note that
not all of them are necessarily applicable to a specific study. A
specific SAP should be consistent with the study objectives, designs,
nature of data, and statistical analysis methods.

1. Primary hypotheses and secondary hypotheses that are consistent with the primary and secondary goals of a study. This also necessarily involves choosing appropriate performance metrics (*see* Subheading 6 for different metrics corresponding to different clinical tasks). For example, the primary goal of an MRMC study might be to show the radiologists using an ML algorithm perform significantly better than without using the algorithm in the task of distinguishing between benign and malignant brain tumors, and a secondary goal might be to show the radiologist using an ML algorithm has significantly better specificity (*S*<sub>p</sub>) at a given sensitivity. Then the null (*H*<sub>0</sub>) and alternative (*H*<sub>1</sub>) primary hypotheses can be stated as

$$
\begin{array}{l}
H_0 : \text{AUC}_{\text{with ML}} = \text{AUC}_{\text{without ML}}; H_1 \\
\qquad : \text{AUC}_{\text{with ML}} > \text{AUC}_{\text{without ML}}.
\end{array}
$$

And the secondary hypotheses may be stated as

$$
H_0 : S_{p_{\text{with ML}}} = S_{p_{\text{without ML}}}; H_1 : S_{p_{\text{with ML}}} > S_{p_{\text{without ML}}}
$$

2. A plan for use of patient data in various stages of ML algorithm development and performance assessment. As discussed in Subheading 3, patient data are used in both the development and assessment of ML algorithms. A pre-specified plan for appropriate use of patient data is crucial for achieving the goals of algorithm development and performance validation and controlling various sources of bias in the process.
3. Methods for analyzing the study data to estimate the pre-specified performance metric, the uncertainty of the metric (e.g., standard deviations and confidence intervals) accounting for all sources of variability including reference standard as needed, and the test statistic for hypothesis testing. It is critically important to examine if the assumptions behind the statistical methods are appropriate for the data and, when necessary, use an alternative method to verify the results.

4. Sample size determination. In a standalone performance assessment study, this is to determine the number of patients to be included in the study such that the study data are representative of the intended patient population (see Subheading 3.3.3) and, when applicable, the study has sufficient statistical power (typically set to be >80%) to detect a significant effect (e.g., superior performance compared with a control). With a single source of variability, standard statistical methods and software tools are often useful for sizing a standalone performance assessment study.

In an MRMC study, both the number of readers and the number of cases need to be determined. Sample size determination is again mainly for assuring a reasonable chance of success in the study planning stage. From a technical point of view (i.e., not taking into consideration practical issues such as budget), sample size is typically determined by considering (1) that the sample sizes are large enough to include samples that represent the intended patient and reader populations and (2) the sample sizes are sufficient to achieve a target statistical power in a hypothesis testing study. Due to the complexity, specialized software tools can be used for sizing an MRMC study [87], and the MRMC software tools cited in Subheading 7 provide the sizing functionality.

The information needed for sizing a pivotal study is often obtained in a pilot study, as discussed in Subheading 3. However, sometimes the pilot study is too limited to provide reliable information, and one may find attempting to re-size a pivotal study after an interim analysis of the data. Naively re-sizing the study based on information obtained in the same study may inflate the type I error rate. Huang et al. [88] developed an approach that allows adaptive re-sizing of an MRMC study with information obtained in an interim analysis such that the statistical power is adjusted to a target value and the type I error rate is retained by paying a statistical penalty in the final hypothesis testing.

5. A plan for adjusting p-values and/or confidence intervals for multiple comparisons or hypothesis tests.

6. A plan for handling missing data and assessing the impact of missing data (e.g., missing reader data, missing follow-up data
to confirm negative results) on the study conclusions. Although statistical techniques may be used to address issues of loss-to-follow-up and missing data, these techniques often employ major assumptions that cannot be fully validated for a particular study. Therefore, the best way to address issues of missing data due to loss-to-follow-up is to plan to minimize its occurrence during the planning and management of the clinical study. Nevertheless, the study protocol should pre-specify appropriate statistical data analysis methods, in addition to sensitivity analyses, for handling missing data.

# 9 Summary Remarks

In this chapter, we provided an overview of a performance assessment framework for imaging-based ML algorithms. We discussed general considerations in study design and data collection, establishment of a reference standard, algorithm documentation, algorithm standalone performance as well as clinical reader studies, and statistical data analysis in performance testing. We believe that these topics are relevant not only in the regulatory setting but also to reproducible science and technology development. Because patient data and clinical experts' annotations are used in both the *development* and *assessment* of ML algorithms, performance assessment should be considered from the very beginning of development to make efficient use of available data. In addition, performance assessment and algorithm development (e.g., tuning, internal validation) are often iterative; meaningful assessment methodologies and tools are not only meant to make the assessment *rigorous* but also *cost-effective*. Furthermore, performance assessment methodologies are also tremendously helpful to assure quality and reproducibility, control bias, and avoid pitfalls and blind alleys.

Machine learning technologies are still rapidly evolving, and their applications in medicine and brain imaging in particular are expanding. It is widely recognized that ML is playing a pivotal role in revolutionizing medicine and promoting public health to a new level. Accompanying these potential developments are new research questions on assessment methodologies. We have touched upon topics in this chapter such as novel types of clinical applications enabled by ML and continuous learning ML. Other exciting topics may include improvement and assessment of robustness and generalizability of ML algorithms, synthetic data augmentation, characterization of bias/fairness, and uncertainty-aware ML algorithms that output not only clinical conditions of interest but also “I don’t know,” among many others. We believe that assessment methodologies and regulatory science play a critical role in fully realizing the great potential of ML in medicine, in facilitating ML device innovation, and in accelerating the translation of these technologies from bench to bedside to the benefit of patients.
## Acknowledgments

The authors thank Drs. Robert Ochs, Alexej Gossmann, and Aldo Badano for carefully reviewing the manuscript and providing helpful comments.

## References

1. Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH, Summers RM, Giger ML (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46(1):e1-e36. https://doi.org/10.1002/mp.13264

2. Lui YW, Chang PD, Zaharchuk G, Barboriak DP, Flanders AE, Wintermark M, Hess CP, Filippi CG (2020) Artificial intelligence in neuroradiology: current status and future directions. Am J Neuroradiol 41(8):E52-E59. https://doi.org/10.3174/ajnr.A6681

3. U.S. Food and Drug Administration (2017) De Novo classification process (Evaluation of Automatic Class III Designation). Guidance for Industry and Food and Drug Administration Staff

4. U.S. Food and Drug Administration (2014) The 510(k) program: evaluating substantial equivalence in premarket notifications [510 (k)]. Guidance for Industry and Food and Drug Administration Staff

5. U.S. Food and Drug Administration (2012) Factors to consider when making benefit-risk determinations in medical device premarket approval and De Novo classifications. Guidance for Industry and Food and Drug Administration Staff

6. U.S. Food and Drug Administration (2018) Benefit-risk factors to consider when determining Substantial equivalence in premarket notifications (510(k)) with different technological characteristics. Guidance for Industry and Food and Drug Administration Staff

7. U.S. Food and Drug Administration (2021) Requests for feedback and meetings for medical device submissions: the Q-submission program. Guidance for Industry and Food and Drug Administration Staff

8. Gallas BD, Chan HP, D'Orsi CJ, Dodd LE, Giger ML, Gur D, Krupinski EA, Metz CE, Myers KJ, Obuchowski NA, Sahiner B, Toldano AY, Zuley ML (2012) Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol 19(4):463-477. https://doi.org/10.1016/j.acra.2011.12.016

9. Hastie T, Tibshirani R, Friedman J (2017) The elements of statistical learning. Series in statistics, 2nd (corrected 12th printing) edn. Springer, New York

10. Chan H-P, Sahiner B, Wagner RF, Petrick N (1999) Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 26(12):2654-2668

11. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster R (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22):2402-2410. https://doi.org/10.1001/jama.2016.17216

12. Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic Press, New York

13. Moons KGM, Altman DG, Reitsma JB, Ioanidis JPA, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162(1):W1-W73. https://doi.org/10.7326/m14-0698

14. Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, Tao D (2019) Exploring representativeness and informativeness for active learning. arXiv:1904.06685

15. Huang S, Jin R, Zhou Z (2014) Active learning by querying informative and representative examples. IEEE Trans Pattern Anal Mach Intell 36:1936-1949

16. Sharma D, Shanis Z, Reddy CK, Gerber S, Enquobahrie A (2019) Active learning technique for multimodal brain tumor segmentation using limited labeled images. In: Wang Q, Milletari F, Nguyen HV et al (eds) Domain adaptation and representation transfer and medical image learning with less labels and imperfect data. Springer International Publishing, Cham, pp 148-156
17. Hao R, Namdar K, Liu L, Khalvati F (2021) A transfer learning-based active learning framework for brain tumor classification. Front Artif Intell 4(61):635766. https://doi.org/10.3389/frai.2021.635766

18. Quionero-Candela J, Sugiyama M, Schwaighofer A, Lawrence N (2009) Dataset shift in machine learning. MIT Press, Cambridge MA

19. Moreno-Torres JG, Raeder T, Alaiz-Rodriguez R, Chawla NV, Herrera F (2012) A unifying view on dataset shift in classification. Pattern Recogn 45(1):521-530. https://doi.org/10.1016/j.patcog.2011.06.019

20. Storkey A (2009) When training and test sets are different: characterizing learning transfer. In: Quionero-Candela J, Sugiyama M, Schwaighofer A, Lawrence N (eds) Dataset shift in machine learning. MIT Press, Cambridge, MA, pp 3-28

21. Goldenberg I, Webb G (2019) Survey of distance measures for quantifying concept drift and shift in numeric data. Knowl Inf Syst 60: 591-615. https://doi.org/10.1007/s10115-018-1257-z

22. Rabanser S, Günnemann S, Lipton ZC (2018) Failing loudly: an empirical study of methods for detecting dataset shift. arXiv:1810.11953

23. Dockès J, Varoquaux G, Poline J-B (2021) Preventing dataset shift from breaking machine-learning biomarkers. arXiv:2107.09947

24. Turhan B (2012) On the dataset shift problem in software engineering prediction models. Empir Softw Eng 17(1):62-74. https://doi.org/10.1007/s10664-011-9182-8

25. U.S. Food and Drug Administration (2007) Guidance for industry and FDA staff: statistical guidance on reporting results from studies evaluating diagnostic tests. vol 2007. U.S Food and Drug Administration, Silver Spring

26. Zhou XH, Obuchowski NA, McClish DK (2002) Statistical methods in diagnostic medicine. Wiley

27. Suresh H, Guttag JV (2021) A framework for understanding sources of harm throughout the machine learning life cycle. arXiv:190110002 [cs, stat]

28. Hooker S (2021) Moving beyond “algorithmic bias is a data problem”. Patterns 2(4):100241. https://doi.org/10.1016/j.patter.2021.100241

29. Guo LL, Pfohl SR, Fries J, Posada J, Fleming SL, Aftandilian C, Shah N, Sung L (2021) Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine.

Appl Clin Inform 12(4):808-815. https://doi.org/10.1055/s-0041-1735184

30. National Academies of Sciences E, Medicine (2019) Reproducibility and replicability in science. The National Academies Press, Washington, DC. https://doi.org/10.17226/25303

31. Ioannidis JPA, Allison DB, Ball CA, Coulibaly I, Cui X, Culhane AC, Falchi M, Furlanello C, Game L, Jurman G, Mangion J, Mehta T, Nitzberg M, Page GP, Petretto E, van Noort V (2009) Repeatability of published microarray gene expression analyses. Nat Genet 41(2):149-155. https://doi.org/10.1038/ng.295

32. Baggerly KA, Coombes KR (2009) Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat 3(4): 1309-1334, 1326

33. Pineau J, Vincent-Lamarre P, Sinha K, Larivière V, Beygelzimer A, d'Alché-Buc F, Fox E, Larochelle H (2020) Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). arXiv:2003.12206

34. Norgeot B, Quer G, Beaulieu-Jones BK, Torkamani A, Dias R, Gianfrancesco M, Arnaout R, Kohane IS, Saria S, Topol E, Obermeyer Z, Yu B, Butte AJ (2020) Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med 26(9):1320-1324. https://doi.org/10.1038/s41591-020-1041-y

35. Mongan J, Moy L, Charles E, Kahn J (2020) Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2(2):e200029. https://doi.org/10.1148/ryai.2020200029

36. El Naqa I, Boone JM, Benedict SH, Goodsitt MM, Chan HP, Drukker K, Hadjiiski L, Ruan D, Sahiner B (2021) AI in medical physics: guidelines for publication. Med Phys 48(9):4711-4714. https://doi.org/10.1002/mp.15170

37. Cruz Rivera S, Liu X, Chan A-W, Denniston AK, Calvert MJ, Darzi A, Holmes C, Yau C, Moher D, Ashrafian H, Deeks JJ, Ferrante di Ruffano L, Faes L, Keane PA, Vollmer SJ, Lee AY, Jonas A, Esteva A, Beam AL, Panico MB, Lee CS, Haug C, Kelly CJ, Yau C, Mulrow C, Espinoza C, Fletcher J, Moher D, Paltoo D, Manna E, Price G, Collins GS, Harvey H, Matcham J, Monteiro J, ElZarrad MK, Ferrante di Ruffano L, Oakden-Rayner L, McCradden M, Keane PA, Savage R, Golub R, Sarkar R, Rowley S, The S-A, Group C-AW, Spirit AI, Group C-AS, Spirit
AI, Group C-AC (2020) Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med 26(9):1351-1363. https://doi.org/10.1038/s41591-020-1037-7

38. Collins G, Moons K (2019) Reporting of artificial intelligence prediction models. Lancet 393:1577-1579. https://doi.org/10.1016/S0140-6736(19)30037-6

39. U.S. Food and Drug Administration (2012) Clinical performance assessment: Considerations for computer-assisted detection devices applied to radiology images and radiology device data - premarket approval (PMA) and premarket notification [510(k)] submissions - Guidance for industry and FDA staff. https://www.fda.gov/media/77642/download. Accessed 31 Oct 2021

40. Warfield SK, Zou KH, Wells WM (2004) Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 23(7):903-921

41. Petrick N, Sahiner B, Armato SG III, Bert A, Correale L, Delsanto S, Freedman MT, Fryd D, Gur D, Hadjiiski L, Huo Z, Jiang Y, Morra L, Paquerault S, Raykar V, Salganicoff M, Samuelson F, Summers RM, Tourassi G, Yoshida H, Zheng B, Zhou C, Chan H-P (2013) Evaluation of computer-aided detection and diagnosis systems. Med Phys 40:087001-087017

42. Steyerberg EW (2019) Overfitting and optimism in prediction models. In: Clinical prediction models. Springer, pp 95-112

43. Akkus Z, Galimzianova A, Hoogi A, Rubin DL, Erickson BJ (2017) Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imaging 30(4): 449-459. https://doi.org/10.1007/s10278-017-9983-4

44. Zhang YJ (1996) A survey on evaluation methods for image segmentation. Pattern Recogn 29(8):1335-1346. https://doi.org/10.1016/0031-3203(95)00169-7

45. Zhang YJ (2001) A review of recent evaluation methods for image segmentation. In: Proceedings of the sixth international symposium on signal processing and its applications (Cat. No.01EX467), 13-16 Aug 2001. vol. 141, pp 148-151. https://doi.org/10.1109/ISSPA.2001.949797

46. Meyer CR, Johnson TD, McLennan G, Aberle DR, Kazerooni EA, Macmahon H, Mullan BF, Yankelevitz DF, van Beek EJR, Armato SG 3rd, McNitt-Gray MF, Reeves AP, Gur D, Henschke CI, Hoffman EA, Bland PH, Laderach G, Pais R, Qing D, Piker C, Guo J,

Starkey A, Max D, Croft BY, Clarke LP (2006) Evaluation of lung MDCT nodule annotation across radiologists and methods. Acad Radiol 13(10):1254-1265

47. Taha AA, Hanbury A (2015) Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging 15(1):29. https://doi.org/10.1186/s12880-015-0068-x

48. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26(3):297-302. https://doi.org/10.2307/1932409

49. Jaccard P (1912) The distribution of the flora in the alpine zone. New Phytol 11(2):37-50

50. Willem (2017) FI/Dice-Score vs IoU. Cross Validated. https://stats.stackexchange.com/questions/273537/f1-dice-score-vs-iou/276144#276144. Accessed 9/29/2021

51. Fenster A, Chiu B (2005) Evaluation of segmentation algorithms for medical imaging. In: 2005 IEEE engineering in medicine and biology 27th annual conference, 17-18 Jan 2006. pp 7186-7189. https://doi.org/10.1109/IEMBS.2005.1616166

52. Tharwat A (2021) Classification assessment methods. Appl Comput Inform 17(1): 168-192. https://doi.org/10.1016/j.aci.2018.08.003

53. Hossin M, Sulaiman MN (2015) A review on evaluation metrics for data classification evaluations. Int J Data Min Knowl Manag Process 5(2):1

54. Obuchowski NA (2003) Receiver operating characteristic curves and their use in radiology. Radiology 229(1):3-8

55. Wagner RF, Metz CE, Campbell G (2007) Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 14(6):723-748

56. Chakraborty DP (2018) Observer performance methods for diagnostic imaging: foundations, modeling, and applications with r-based examples. Imaging in medical diagnosis and therapy. CRC Press, Boca Raton, FL

57. ICRU (2008) Receiver operating characteristic analysis in medical imaging. Report 79. International Commission of Radiation Units and Measurements, Bethesda, MD

58. He X, Frey E (2009) ROC, LROC, FROC, AFROC: an alphabet soup. J Am Coll Radiol 6(9):652-655

59. Bunch PC, Hamilton JF, Sanderson GK, Simmons AH (1977) A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE 127: 124-135
60. Edwards DC, Kupinski MA, Metz CE, Nishikawa RM (2002) Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys 29(12):2861-2870. https://doi.org/10.1118/1.1524631

61. Chakraborty DP (2006) Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 13(10): 1187-1193

62. Chakraborty DP (2006) A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol 51(14):3449-3462

63. Padilla R, Netto SL, Silva EABd (2020) A survey on performance metrics for object-detection algorithms. In: 2020 international conference on systems, signals and image processing (IWSSIP), 1-3 July 2020. pp 237-242. https://doi.org/10.1109/IWSSIP48289. 2020.9145130

64. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303-338. https://doi.org/10.1007/s11263-009-0275-4

65. ImageNet (2017) ImageNet object localization challenge. Kaggle. https://www.kaggle.com/c/imenet-object-localization-chal lenge/. Accessed 10/22/2021 2021

66. Liu Z, Bondell HD (2019) Binormal precision-recall curves for optimal classification of imbalanced data. Stat Biosci 11(1):141-161. https://doi.org/10.1007/s12561-019- 09231-9

67. Sahiner B, Chen W, Pezeshk A, Petrick N (2016) Semi-parametric estimation of the area under the precision-recall curve. In: SPIE medical imaging. International Society for Optics and Photonics, pp 97870D-97870D-97877

68. Thompson E, Levine G, Chen W, Sahiner B, Li Q, Petrick N, Samuelson F (2022) Wait-time-saving analysis and clinical effectiveness of computer-aided triage and notification (CADt) devices based on queueing theory. In: Taylor-Phillips CRM-TaS (ed) Medical imaging 2022: Image perception, observer performance, and technology assessment, San Diego, CA, SPIE, p accepted

69. U.S. Food and Drug Administration (2019) Proposed regulatory framework for modifications to Artificial Intelligence/Machine Learning (AI/ML)-based Software as a Medical Device (SaMD) – Discussion paper and request for feedback. U.S Food and Drug Administration. https://www.fda.gov/files/ medical%20devices/published/US-FDA-

Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf. Accessed 31 Oct 2021

70. Feng J, Emerson S, Simon N (2021) Approval policies for modifications to machine learning-based software as a medical device: a study of bio-creep. Biometrics 77(1):31-44. https:// doi.org/10.1111/biom.13379

71. Pennello G, Sahiner B, Gossmann A, Petrick N (2021) Discussion on “approval policies for modifications to machine learning-based software as a medical device: a study of bio-creep” by Jean Feng, Scott Emerson, and Noah Simon. Biometrics 77(1):45-48. https://doi.org/10.1111/biom.13381

72. Dorfman DD, Berbaum KS, Metz CE (1992) Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Investig Radiol 27(9):723-731

73. Obuchowski NA, Rockette HE (1995) Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests an ANOVA approach with dependent observations. Commun Stat Simul Comput 24(2): 285-308. https://doi.org/10.1080/ 03610919508813243

74. Beiden SV, Wagner RF, Campbell G (2000) Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol 7(5): 341-349

75. Gallas BD (2006) One-shot estimate of MRMC variance: AUC. Acad Radiol 13(3): 353-362

76. Hillis SL, Berbaum KS, Metz CE (2008) Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol 15(5): 647-661

77. Gallas BD, Bandos A, Samuelson FW, Wagner RF (2009) A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators. Commun Stat Theory Methods 38(15):2586-2603. https://doi.org/10.1080/03610920802610084

78. Gallas BD, Pennello GA, Myers KJ (2007) Multireader multicase variance analysis for binary data. J Opt Soc Am A 24(12):B70-B80

79. Metz CE (1995) The Dorfman/Berbaum/Metz method for testing the statistical significance of ROC differences: validation studies with continuously-distributed data. The Far-west image perception conference to be given October 13, 1995 in Philadelphia, PA

80. Chen W, Wunderlich A, Petrick N, Gallas BD (2014) Multireader multicase reader studies
with binary agreement data: simulation, analysis, validation, and sizing. J Med Imaging (Bellingham) 1(3):031011-031011. https://doi.org/10.1117/1.JMI.1.3.031011

81. Obuchowski NA (2009) Reducing the number of reader interpretations in MRMC studies. Acad Radiol 16(2):209-217

82. Obuchowski NA, Gallas BD, Hillis SL (2012) Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol 19(12):1508-1517. https://doi.org/10.1016/j.acra.2012.09.012

83. Chen W, Gong Q, Gallas BD (2018) Paired split-plot designs of multireader multicase studies. J Med Imaging (Bellingham) 5(3): 031410. https://doi.org/10.1117/1.JMI.5. 3.031410

84. U.S. Food and Drug Administration (2020) Clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data in premarket notification (510(k))

submissions. Guidance for Industry and Food
and Drug Administration Staff

85. Dwork C, Feldman V, Hardt M, Pitassi T, Reingold O, Roth A (2015) The reusable hold-out: preserving validity in adaptive data analysis. Science 349(6248):636-638

86. Gossmann A, Pezeshk A, Wang Y-P, Sahiner B (2021) Test data reuse for the evaluation of continuously evolving classification algorithms using the area under the receiver operating characteristic curve. SIAM J Math Data Sci 3: 692-714. https://doi.org/10.1137/20M1333110

87. Hillis SL, Obuchowski NA, Berbaum KS (2011) Power estimation for multireader ROC methods an updated and unified approach. Acad Radiol 18(2):129-142

88. Huang Z, Samuelson F, Tcheuko L, Chen W (2020) Adaptive designs in multi-reader multicase clinical trials of imaging devices. Stat Methods Med Res 29(6):1592-1611. https://doi.org/10.1177/0962280219869370

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
**Chapter 24**

**Main Existing Datasets for Open Brain Research on Humans**

Baptiste Couvy-Duchesne ID, Simona Bottani, Etienne Camenen, Fang Fang, Mulusew Fikere ID, Juliana Gonzalez-Astudillo ID, Joshua Harvey ID, Ravi Hassanaly, Irfahan Kassam ID, Penelope A. Lind ID, Qianwei Liu, Yi Lu, Marta Nabais, Thibault Rolland, Julia Sidorenko, Lachlan Strike ID, and Margie Wright ID

**Abstract**

Recent advances in technology have made possible to quantify fine-grained individual differences at many levels, such as genetic, genomics, organ level, behavior, and clinical. The wealth of data becoming available raises great promises for research on brain disorders as well as normal brain function, to name a few, systematic and agnostic study of disease risk factors (e.g., genetic variants, brain regions), the use of natural experiments (e.g., evaluate the effect of a genetic variant in a human population), and unveiling disease mechanisms across several biological levels (e.g., genetics, cellular gene expression, organ structure and function). However, this data revolution raises many challenges such as data sharing and management, the need for novel analysis methods and software, storage, and computing.

Here, we sought to provide an overview of some of the main existing human datasets, all accessible to researchers. Our list is far from being exhaustive, and our objective is to publicize data sharing initiatives and help researchers find new data sources.

**Key words** Genetic, Methylation, Gene expression, Brain MRI, PET, EEG/MEG, Omics, Electronic health records, Wearables

**1 Aims**

We sought to provide an overview and short description of some of the main existing human datasets accessible to researchers. We hope this chapter will help publicize them as well as encourage the sharing of datasets for open science. As much as possible, we tried to provide practical aspects, such as data type, file size, sample demographics, study design, as well as links toward data use/transfer agreements. We hope this can help researchers study larger and more diverse data, in order to advance scientific discovery and improve reproducibility.
This chapter does not aim to provide an exhaustive list of the dataset and data types currently available. In addition, the interested readers may refer to the complementary chapters that focus on data processing, feature extraction, and existing methods for their analyses.

## 2 Introduction

The availability of data used in research is one of the cornerstones of open science, which contributes to improving the quality, reproducibility, and impact of the findings. In addition, data sharing increases openness and transparent and collaborative scientific practices. The global push for open science is exemplified by the recent publication of UNESCO guidelines [1], the engagement of many research institutions, and the requirements of some scientific journals to make data available upon publication. Finally, the sharing and re-use of data also maximizes the return on investment of the agencies (e.g., states, charities, associations) that fund the data collection.

In light of this, our chapter aims at providing a broad (albeit partial) overview of some of the human datasets publicly available to researchers. To assist researchers and data managers, we first describe the different file formats and the size of the different data types (see Table 1). As many of these data are high-dimensional, the size of the data can cause storage and computational challenges, which need to be anticipated before download and analysis. Of note, some datasets cannot be downloaded or analyzed outside of a dedicated system/server. This is the case of the UK Biobank (UKB) exome and whole genome sequencing, whose sheer size has led to the creation of a dedicated Research Analysis Platform, accessible (at some cost) by UKB-approved researchers. In addition, the Swedish registry data is only accessible via national dedicated servers due to the extreme sensitive nature of the data.

This chapter breaks down into sections that focus on each data type, although the same dataset may be mentioned in several sections. Beyond a practical writing advantage (each author or group of authors contributed a section), this also reflects the fact that most datasets are organized around a central data type. For example, the ADNI (Alzheimer's disease Neuroimaging Initiative) focuses on brain imaging and later included genotyping information. Another example is the UKB, which released genotyping data of the 500 K participants in 2017, is now collecting brain MRI (as well as cardiac and abdominal MRI, whole-body DXA, and carotid ultrasound), and has recently made available sequencing data. The different sections also discuss and present the specific data sharing tools and portals (e.g., LONI for brain imaging, GTEx for gene expression) or organization of the different fields (e.g., consortia in
<table><caption>**Table 1**
Overview of data types and sample size</caption><thead><tr><th>Data</th><th>Subtype</th><th>File type/extension</th><th>Description</th><th>Approx. size of one sample</th></tr></thead><tbody><tr><td rowspan="2">Clinical</td><td>Self-reports</td><td>Text</td><td rowspan="2"></td><td>~100 KB</td></tr><tr><td>EHR</td><td>Text</td><td>~100 KB</td></tr><tr><td rowspan="2">Neuroimaging</td><td>MRI</td><td>NIfTI (.nii/.nii.gz) or DICOM (.dcm)</td><td>3D image (.nii)<br/>2D slices (.dcm) that compose the 3D volume</td><td>T1w ~50 MB<br/>T2 FLAIR ~30 MB<br/>SWI ~30 MB<br/>DWI ~400 MB (highly variable depending on sequence)<br/>fMRI ~500 MB (100 MB per minute of acquisition)</td></tr><tr><td></td><td></td><td></td><td>~1 GB after processing</td></tr><tr><td rowspan="3">PET</td><td></td><td>NIfTI (.nii / .nii.gz) or DICOM (.dcm)</td><td>3D image (.nii) 2D slices (.dcm) that compose the 3D volume</td><td>~15 MB</td></tr><tr><td>EEG</td><td>.edf, .gdf, .eeg, .csv, .mat</td><td>Raw EEG signal (.egg, .gdf, .edf – file formats), processed/formatted data (.csv, .mat)</td><td>~15 MB</td></tr><tr><td>MEG</td><td>.fif, .bin, .csv</td><td>Raw MEG signal (.fif, .bin) processed/formatted data (.csv, .mat)</td><td>~50 MB</td></tr><tr><td rowspan="5">Genetics</td><td>Twin/family sample</td><td>Text</td><td>Phenotypes and pedigree information</td><td>~10 KB</td></tr><tr><td>Genotyping</td><td>.bgen, .bed (.bim + .bed + .fam)</td><td>Binary files</td><td>~5 MB</td></tr><tr><td>GWAS</td><td>Text</td><td>Effect size, test statistic, p-value, effect allele from association testing</td><td>Not individual-level data, ~500 MB for a genome-wide association summary</td></tr><tr><td>Exome sequencing</td><td>.cram, .bcf, .vcf</td><td>Each variant site with multiple sequence quality metrics and trained machine learning filters to identify and exclude inconsistencies. Followed by initial QC with various parameters and thresholds</td><td>~1 M</td></tr><tr><td>Whole genome sequencing</td><td>.cram, .bcf, .vcf</td><td>Curated using TOPMed variant calling algorithm. All freezes are fully processed</td><td>~5 MB</td></tr></tbody></table>

(continued)
<table><caption>Table 1
(continued)</caption><thead><tr><th>Data</th><th>Subtype</th><th>File type/extension</th><th>Description</th><th>Approx. size of one sample</th></tr></thead><tbody><tr><td rowspan="2">Genomics</td><td>Methylation</td><td>.ids<br>.txt or .csv.</td><td>Raw binary intensity file (two per sample, one for green and red channels)<br>Processed, normalized, and filtered beta matrix</td><td>~16 MB (450 K)<br>~27 MB (EPIC)<br>~8 MB (450 K)<br>~14 MB (EPIC)</td></tr><tr><td>Expression</td><td>.bed (.bim + .bed + .fam)<br>Text</td><td>Fully processed, filtered, and normalized gene expression matrices (in BED format) for each tissue<br>Covariates used in eQTL analysis. Includes genotyping principal components and PEER factors</td><td>~1 MB per individual<br>~6 KB</td></tr><tr><td>Smartphone and sensors</td><td>Actigraphy</td><td>Varies with product (e.g., GENEActiv uses .bin)</td><td>Raw accelerator data</td><td>~500 MB (GENEActiv, 14 days of recording)</td></tr></tbody></table>

Size of one sample may vary based on the technology used to generate the data (e.g., MRI resolution, genotyping chip)
genetics). Every time, we have tried to include the largest dataset (s) available, as well as the commonly used ones, although the selection may be subjective and reflect the authors’ specific interests (e.g., age or disease groups).

All datasets are listed in a single table (see Table 2), which includes information about country of origin, design (e.g., cross-sectional, longitudinal, clinical, or population sample), and age range of the participants. Unless specified, the datasets presented include male and female participants, although the proportion may differ depending on the recruitment strategy and disease of interest. In addition, the table lists (and details) the different data types that have been collected on the participants. We have only focused on a handful of data types: genetic data (including twin/family samples, genotyping, and exome and whole genome sequencing), genomics (methylation and gene expression), brain imaging (MRI and PET), EEG/MEG, electronic health records (hospital data and national registry), as well as wearable and sensor data. However, we have included additional columns “Other omics” and “Specificities” that list other types of data being collected, such as proteomics, metabolomics, microRNA, single-cell sequencing, microbiome, and non-brain imaging.

Our main table (see Table 2) also includes the URLs to the dataset websites and data transfer/agreement. From our experience, data access can take between an hour and up to a few months. The agreements almost always require a review of the project and to acknowledge the data collection team and funding sources (e.g., under the form of a byline, a paragraph in the acknowledgment, and more rarely co-authorships). Standard restrictions of use include that the data cannot be redistributed and that the users do not attempt to identify participants. Specific clauses are often added depending on the nature of data and the specific laws and regulations of the countries it originates from.

There is a growing scientific and ethical discussion about the representativity of the datasets being used in research. Researchers should be aware of the biases present in some datasets (e.g., “healthy bias” in the UKB [2]), which should be taken into account in study design (e.g., analysis of diverse ancestry being collected in genetics [3]), when reporting results [2, 4] and evaluating algorithms [5, 6]. Overall, our (selected) list exemplifies the need for datasets from under-represented countries or groups of individuals (e.g., disease, age, ancestry, socioeconomic status) [7, 8]. Our main table (see Table 2) will be accessible online, in a user-friendly, searchable version. Finally, we will also make this table collaborative (via GitHub https://github.com/baptisteCD/ MainExistingDatasets) in order to grow this resource beyond this book chapter.
<table><caption>Table 2
Description of a selection of the main human datasets available for research</caption><thead><tr><th>Sample</th><th>N</th><th>Age range</th><th>Country</th><th>Cross-sectional/<br>longitudinal</th><th>Clinical</th><th>Neuroimaging<br>(MRI, PET, fMRI)</th><th>EEG/MEG</th><th>Genetics (genotyping,<br>exome, WGS,<br>tions)</th><th>Genomics</th><th>Smartphones<br>and sensors</th><th>Other omics</th><th>Website/Data<br>Transfer<br>Agreement</th><th>Reference<br>article(s)</th><th>Specificities</th></tr></thead><tbody><tr><td>Alzheimer's<br>Disease<br>Neurodegenerative<br>Initiative<br>(ADNI)</td><td>2215</td><td>55-90</td><td>USA and<br>Canada</td><td>Longitudinal (up<br>to 9 years of<br>follow-up,<br>ongoing)</td><td>Neurological<br>(Alzheimer's),<br>cognition, lumbar<br>puncture</td><td>MRI (T1w, T2, DWI,<br>stMRI)<br>PET ("E"-FDG,<br>FBB, AV45,<br>FB)</td><td>NA</td><td>Genotyping,<br>WGS</td><td>Methylation<br>(subset<br>of 653<br>individuals—<br>3 time<br>points)</td><td>NA</td><td>Transcriptomics,<br>CFN<br>proteomics,<br>metabolomics,<br>lipidomics</td><td>http://adl-loni.unc.edu/data-samples/access-data/</td><td>[11, 191]</td><td>Four waves<br>(ADNI 2,<br>GO, 3) with<br>different inclusion<br>and protocols</td></tr><tr><td>Australian<br>Imaging<br>Biomarkers<br>and Lifestyle<br>Study of Aging<br>(AIBL)</td><td>726</td><td>60+</td><td>Australia</td><td>Longitudinal (up<br>to 6 years of<br>follow-up)</td><td>Neurological<br>(Alzheimer's),<br>cognition, lumbar<br>puncture</td><td>MRI (T1w, T2,<br>DWI, stMRI)<br>PET (FB, AV45,<br>Flare)</td><td>NA</td><td>Genotyping</td><td>Methylation</td><td>ActiGraph activity NA<br>(10% of<br>sample)</td><td>https://adl-loni.unc.edu/collaboration/access/appLicense.jsp<br>https://aibl.cairo.au/</td><td></td><td>[12]</td><td>Neuroimaging and<br>selected clinical<br>data available via<br>the LONI platform<br>Full sample (N = 1200),<br>including extended<br>clinical,<br>genotyping, or<br>methylation<br>available via<br>application to<br>CSIRO</td></tr><tr><td>Open Access<br>Series of<br>Imaging<br>Studies 1-3<br>(OASIS)</td><td>1096</td><td>42-95</td><td>USA</td><td>Longitudinal (up<br>to 12 years of<br>follow-up)</td><td>Neurological<br>(Alzheimer's),<br>cognition</td><td>MRI (T1w, T2,<br>FLAIR, ASL,<br>SWI, time<br>of flight,<br>stMRI, and<br>DWI)<br>PET (FB, AV45,<br>FG)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://www.oasis-brains.org<br>https://www.oasis-brains.org/#access</td><td></td><td>[13]</td><td>Retrospective dataset<br>from imaging<br>projects collected<br>by WUSTL Knight<br>ADBRC over 30<br>years<br>Two other<br>(non-independent)<br>datasets available:<br>OASIS1 and<br>OASIS2</td></tr><tr><td>Adolescent<br>Brain<br>Cognitive<br>Development<br>(ABCD)</td><td>-11,878</td><td>9-12</td><td>USA</td><td>Longitudinal (up<br>to 2 years of<br>follow-up,<br>ongoing)</td><td>Self- and parental<br>rating. Substance<br>use, mental health<br>(psychiatry),<br>cognition,<br>physical health</td><td>MRI (T1w, T2,<br>stMRI, tMRI)</td><td>NA</td><td>Genotyping,<br>pedigree<br>(twanning)</td><td>Methylation</td><td>iPAD tasks<br>and testing</td><td>NA</td><td>https://abcstudy.org<br>https://nda.nih.gov/<br>abcod/request-access</td><td></td><td>[16, 17]</td><td>Objective: 10 years of<br>follow-up</td></tr><tr><td>Enhancing<br>Neuroimaging<br>Grades<br>through Meta-<br>analysis<br>(ENIGMA)</td><td>&gt;50,000<br>indiv.<br>incl.: 9572<br>(SCZ),<br>6503 (BD),<br>10,105<br>(MDD),<br>1868<br>(PTSD),<br>3240<br>(SUD),<br>3665<br>(OCD),<br>4180<br>(ADHD),<br>18,605 (life<br>span)</td><td>3-90 (no<br>restriction)</td><td>Worldwide<br>(43+<br>countries)</td><td>Cross-sectional<br>and<br>longitudinal</td><td>Psychiatry, neurology,<br>addiction,<br>suicidality,<br>brain injury, HIV,<br>antisocial<br>behavior</td><td>T1w, DWI,<br>stMRI,<br>tMRI</td><td>Resting-state<br>EEG</td><td>Genotyping, CNYs,<br>pedigree<br>(twanning)</td><td>Methylation</td><td>NA</td><td>NA</td><td>https://enigma.ini.unc.edu/poin/<br>http://enigma.ini.unc.edu/research/download-enigma-gwas-results/</td><td>Neuroimaging<br>projects<br>[27<br>[112, 114]</td><td>Consortium organized<br>around genetics<br>and/or disease/<br>trait working<br>groups as well as<br>non-clinical<br>working groups<br>with focus on sex,<br>healthy aging,<br>plasticity, etc.<br>Imaging and genetic<br>protocols and<br>genome-wide<br>association statistics<br>are available for<br>download on the<br>ENIGMA website</td></tr></tbody></table>
<table><tr><td>ChNose Brain PET 116<br>Template<br>(CNPET)</td><td>27-81</td><td>China</td><td>Cross-sectional</td><td>Healthy subjects</td><td>PET (¹³F-FDG)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://www.wuaitrc.org/<br>doi/landing/page.<br>php?table=groups&amp;<br>id=14860&doi=<br>https://www.wuaitrc.org/<br>projects/cnpet</td><td>39</td><td>Data made to build a<br>Chinese-specific<br>SPM PET brain<br>template</td></tr><tr><td>Centre for<br>Integrated<br>Molecular<br>Brain Imaging<br>(CIMBI)</td><td>-2000</td><td>17-93</td><td>Denmark</td><td>Cross-sectional</td><td>Mean and physical<br>state, personality,<br>and background<br>Neuropsychological<br>measures<br>(memory,<br>language, etc.)</td><td>MRI (T1w, T2w,<br>DWI, fMRI)<br>PET (¹¹C-5-HT)</td><td>NA</td><td>NA</td><td>Genetic<br>polymorphisms<br>relevant for the<br>5-HT system</td><td>NA</td><td>NA</td><td>NA</td><td>https://www.cimbi.dk/<br>index.php</td><td>38</td><td>Data sharing only<br>within the<br>European Union<br>(EU). Visiting<br>access for non-EU<br>researchers</td></tr><tr><td>Moor Imagery<br>dataset from<br>Cho et al.<br>[192]</td><td>52</td><td>24.8 SD =<br>3.86 years old</td><td>South<br>Korea</td><td>Cross-sectional</td><td>Healthy subjects</td><td>NA</td><td>NA</td><td>EEG<br>(64<br>channels;<br>real hand<br>movements,<br>motor<br>imagination<br>hand<br>movement<br>tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>http://noabb.<br>neurotoch.com/<br>docs/generated/<br>noabb generateds.<br>Cho.2017.<br>hami#noabb.<br>daets Cho.2017</td><td>192</td><td>BCI experiments</td></tr><tr><td>EEG Alpha Waves<br>dataset</td><td>20</td><td>19-44 years old</td><td>France</td><td>Cross-sectional</td><td>Healthy subjects</td><td>NA</td><td>NA</td><td>EEG<br>(16<br>channels;<br>resting state,<br>eyes open/<br>closed</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://xenodo.org/<br>record/2608110#<br>YIC154AZZIE</td><td>193</td><td>Alpha waves dataset.<br>BCI experiments</td></tr><tr><td>Multiper spectra<br>recorded<br>daring<br>GABAergic<br>anesthetic<br> unconsciousness</td><td>55</td><td></td><td>USA</td><td>Cross-sectional</td><td>Healthy participants<br>and patients<br>receiving an<br>anesthesia care in<br>an operating<br>room context</td><td>NA</td><td>NA</td><td>EEG (64 for<br>healthy<br>volunteers, 6<br>for patients<br>under<br>anesthesia)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://physispect.org/<br>content/ccg-<br>power-<br>anesthesia/1.00/</td><td>194</td><td>Patients under<br>anesthesia<br>during surgery</td></tr><tr><td>A multi-subject,<br>multi-modal<br>human<br>neuroimaging<br>dataset</td><td>19 (16 for<br>MEG and<br>EEG, 3 for<br>fMRI)</td><td>23-37 years old</td><td>UK</td><td>Longitudinal (2<br>visits over 3<br>months)</td><td>Healthy subjects</td><td>fMRI</td><td>NA</td><td>MEG, EEG (70<br>channels;<br>resting state,<br>sensory-<br>motor tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://legacy.opentinn.<br>org/dataset/<br>ds000117/</td><td>195</td><td>OpenMRI database,<br>accession number<br>ds000117</td></tr><tr><td>Moor imagery,<br>suited<br>classification<br>application</td><td>10</td><td>26-46 years old</td><td>Germany</td><td>Cross-sectional</td><td>Healthy subjects</td><td>NA</td><td>NA</td><td>EEG (59<br>channels;<br>hands, feet,<br>and tongue<br>motor<br>imagination<br>tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>http://www.bbci.de/<br>competition/iv/<br>desc_1.html</td><td>196</td><td>Data used for a BCI<br>competition. Also<br>include simulated/<br>synthetic data</td></tr><tr><td>BCI Competition<br>2008: Goetz<br>data set A</td><td>9</td><td>NA</td><td>Austria</td><td>Cross-sectional</td><td>Healthy subjects</td><td>NA</td><td>NA</td><td>EEG (22<br>channels;<br>hands, feet,<br>and tongue<br>motor<br>imagination<br>tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>http://www.bbci.de/<br>competition/iv/<br>desc_2a.pdf</td><td>197</td><td>Data used for a BCI<br>competition</td></tr></table>

(continued)
<table><caption>Table 2
(continued)</caption><thead><tr><th>Sample</th><th>N</th><th>Age range</th><th>Country</th><th>Cross-sectional/ longitudinal</th><th>Clinical</th><th>Neuroimaging (MRI, PET MRI)</th><th>EEG/MEG</th><th>Genetics (genotyping, exome, WES, twins)</th><th>Genomics</th><th>Smartphones and sensors</th><th>Other omics</th><th>Website/Data Transfer Agreement</th><th>Reference article(s)</th><th>Specificities</th></tr></thead><tbody><tr><td>BCI Competition 2008/2009: Genz data set B</td><td>9</td><td>NA</td><td>Austria</td><td>Cross-sectional</td><td>Healthy subjects</td><td>NA</td><td>EEG (3 channels); hands motor imagination tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>http://www.bci.de/ competition/iv/ disc_2b.pdf</td><td>[197]</td><td>Data for a BCI competition</td></tr><tr><td>EEG data from Monnens et al. [198]</td><td>64 (34 MDD, 30 healthy) years old</td><td>Malaysia</td><td>Longitudinal (multiple visits to the clinic)</td><td>Case-control for major depressive disorder</td><td>NA</td><td>EEG (19 channels)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://journals.plos.org/plosone/ article/10.1371/journal.pone.0171409</td><td>[198]</td><td>EEG (reasing task), MDD based on medical history of patients</td></tr><tr><td>EEG data for ADHD/control children</td><td>121</td><td>7-12 years old</td><td>South Korea</td><td>Cross-sectional</td><td>ADHD (61 children) and healthy controls [61]</td><td>NA</td><td>EEG (19 channels); visual attention tasks</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://icee-dataport.org/open-access/ ceg-data-adhd-control-children</td><td>[199]</td><td>Visual cognitive tests on children with ADHD, using videos</td></tr><tr><td>EEG and EOG data from Juramilla-Gonzales et al. [61]</td><td>NA</td><td>Germany</td><td>Longitudinal (2 to 10 visits)</td><td>Amytrophic lateral sclerosis, locked-in state</td><td>NA</td><td>EEG (116 channels)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://doi.org/10. 6084/m9figharc. 13148762</td><td>[200]</td><td>Spelling task with eye movement-based answers</td></tr><tr><td>Temple University Hospital (TUH) EEG Corp</td><td>10,874</td><td>1-90</td><td>USA</td><td>Cross-sectional</td><td>Epilepsy, stroke, concussion, healthy</td><td>NA</td><td>EEG (20-41 channels)</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://sippiopress.com/projects/tuh- ceg/html/ downloads.shtml</td><td>[201]</td><td>Extensive dataset of 30 K clinical EEG recordings collected at TUH from 2002 to today: Epilepsy subset: TUEP dataset</td></tr><tr><td>Bern-B Barcelona EEG database</td><td>5</td><td>NA</td><td>Spain</td><td>Longitudinal</td><td>Epilepsy</td><td>NA</td><td>EEG (64 channels); rest</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://repository.upf.edu/handle/ 10230/42829</td><td>[202]</td><td>Intracranial EEG samples before and after surgery</td></tr><tr><td>CHB-MIT Salp EEG Database</td><td>22</td><td>1.5-22</td><td>USA</td><td>Longitudinal</td><td>Seizures and intracranial seizures</td><td>NA</td><td>EEG (21 channels); resting state</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>https://www.physionet.org/content/chbmc/1.0.0/</td><td>[203]</td><td>Pediatric subjects with intracranial seizures</td></tr><tr><td>Brisa/Narul- Computer Intervention (BNCI) Horizon</td><td>2</td><td>NA</td><td>Austria</td><td>Longitudinal</td><td>Chronic stroke</td><td>NA</td><td>EEG (2 channels); eye tracking task</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>http://nчио- horizon-2020.ca/ database/data-sets</td><td>[204]</td><td>The BNCI is an open-source project with several datasets. Stroke is 66. SCT training in stroke (006-2014)*</td></tr><tr><td>Queensland Twin Addict/Adolescent Brain (QZAB)</td><td>422 (baseline)</td><td>9-14 (baseline)</td><td>Australia</td><td>Longitudinal</td><td>Population sample: Parental and/or self-report mental health, cognition, and social behavior measures</td><td>MRI (T1w, T2w, FLAIR, DWI, stMRI, tMRI, ASL)</td><td>NA</td><td>Pedigree (twin/ sibling), genotyping</td><td>NA</td><td>Wrist-worn accelerometer</td><td>Glut microsome</td><td>https:// imaginggenomics. net.au/</td><td>[205]</td><td>Includes participants from Queensland Twin Registry and Twin Research Australia</td></tr></tbody></table>
<table><thead><tr><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td>Queensland Twin Imaging Study (QITS)</td><td>1200-1230</td><td>Australia</td><td>Cross-sectional</td><td>Population sample (as part of BATS). Self-report, mental health, cognition, substance use, and personality measures</td><td>MRI (T1w, DWI, stMRI, tMRI)</td><td>NA</td><td>Pedigree (twin/ sibling), genotyping</td><td>NA</td><td>NA</td><td>NA</td><td>https://imaginggenomics.net.au/</td><td>[206]</td><td>Include participants from Brisbane adolescent twin study (BATS)</td></tr><tr><td>Human Contraceptive Project, young adults (HCP-T2A)</td><td>-1200</td><td>22-35</td><td>USA</td><td>Population sample. Self-report, mental health, cognition, personality, and substance use measures</td><td>MRI (T1w, T2w, DWI, stMRI, tMRI)</td><td>MEG (n = 95)</td><td>Pedigree (twin/ sibling), Genotyping</td><td>NA</td><td>NA</td><td>NA</td><td>https://dth.humanconnect.org</td><td>[29]</td><td>Expanded to development and aging projects (similar imaging protocols, but extensions do not include twins)</td></tr><tr><td>Verum Era Twin Study of Aging (FEZTSA)</td><td>1237 (baseline; 500+ with MRI)</td><td>51-60 (baseline)</td><td>USA</td><td>US veterans, males only</td><td>MRI (T1w, DWI, ASL (subset of sample))</td><td>NA</td><td>Pedigree (twin), Genotyping</td><td>NA</td><td>NA</td><td>NA</td><td>https://neckschul.ac.il/son/psychoiary/research/VEITSA/Researchers/ Page/ default.aspx</td><td>[83]</td><td>Subset of the Vietnam Era Twin Registry, males only</td></tr><tr><td>Older Australian Twin Study (OATS)</td><td>623 (baseline) >65</td><td>Australia</td><td>Longitudinal</td><td>Population sample. Self-report, medical and mental health and neurology, chological measures</td><td>MRI (T1w, DWI, tMRI; PET)</td><td>NA</td><td>Pedigree (twin/ sibling), Genotyping</td><td>NA</td><td>NA</td><td>NA</td><td>https://cheba.unsw.edu.au/research-projecs/older-australian-twin-study</td><td>[84]</td><td>Recruited through the Australian Twin Registry</td></tr><tr><td>Swedish Twin Registry (STR)</td><td>87,000 twin pairs</td><td>All ages</td><td>Sweden</td><td>Longitudinal (since the late 1950s)</td><td>NA</td><td>NA</td><td>Genome-wide single-nucleotide polymorphism array genotyping</td><td>Methylation is not available in the STR yet but is available in the Swedish adoption/ twin study of aging (SAYTSA), a sub-study of the STR</td><td>NA</td><td>NA</td><td>https://ki.ac/en/research/the-swedish-twin-registry</td><td>[74]</td><td>Twin Registry</td></tr><tr><td>UK Rijkbank (UKB)</td><td>-502,000 -50,000 (imaging subset) -100,000 (actigraphy subset)</td><td>37-73 44-82 (imaging) 40-69 (actigraphy)</td><td>UK</td><td>Longitudinal</td><td>Self-reported and EHR medical history (incl. cancer, neurology, COVID-19)</td><td>MRI (T1w, T2w, FLAIR, DWI, SWI, stMRI, tMRI)</td><td>NA</td><td>Genotyping, crome, WGS, pedigree</td><td>NA</td><td>Actigraphy</td><td>Microbiology (future release)</td><td>https://ibham.naphox.ac.uk/ applications</td><td>Population based sample (volunteers), healthy bias, ongoing resource and data collection/ target MRI sample 100 K, retest 10 K. also available: Cardiac and abdominal MRI, whole-body DXA (dual-energy X-ray) absorptionometry.</td></tr></tbody></table>

(continued)
<table><caption>Table 2
(continued)</caption><thead><tr><th>Sample</th><th>N</th><th>Age range</th><th>Country</th><th>Cross-sectional/ longitudinal</th><th>Clinical</th><th>Neuroimaging (MRI, PET MRI)</th><th>EEG/MEG</th><th>Genetics (genotyping, exome, WGS, twins)</th><th>Genomics</th><th>Smartphones and sensors</th><th>Other omics</th><th>Website/Data Transfer Agreement</th><th>Reference article(s)</th><th>Specifics</th></tr></thead><tbody><tr><td>Avon Longitudinal Study of Parents and Children (ALSPAC): Accessible Resource for Integrated Epigenomic Studies (ARIES)</td><td>2044</td><td>Average age: Mothers (antenatal = 28.7, follow-up = 47.5), offspring (birth = 40 weeks, childhood = 7.5, adolescence = 17.1)</td><td>UK</td><td>Longitudinal (2 time points for mother, 3 offspring)</td><td>Clinical evaluations, obstetric data, cognition, questionnaires</td><td>MRI (T1w, DWI, mDESPOT (subset of offspring cohort))</td><td>NA</td><td>Genotyping</td><td>Methylation</td><td></td><td>Transcriptomics</td><td>http://www.ariesepigenomics.org/ak/https://github.com/MRCIEU/aries</td><td>[207]</td><td>General population study (health and development); following 1022 mother-offspring pairs; 2 time points for mother, 3 for offspring</td></tr><tr><td>Boombank-based Integrative Omics Studies (The BIOS Consortium)</td><td>-4000</td><td>18-87</td><td>Netherlands</td><td>Longitudinal</td><td>Clinical information and bioassays (depending on the sub-cohort)</td><td>NA</td><td>NA</td><td>Genotyping, pedigree</td><td>Methylation</td><td></td><td>Transcriptomics, metabolomics</td><td>https://www.bhuni.nl/ node/24</td><td>[208]</td><td>Population study including various sub-cohorts, covering differing research designs (lifecycles, Leiden longer-term study, Netherlands twin registry, Rotterdam study, CODAM, and the prospective ALS study Netherlands); access via European genome-phenome archive (EGA) and SURFsaera high performance computing cloud</td></tr><tr><td>Frankfurt Heart 4241 Study</td><td></td><td>Offspring cohort mean age = 66, third-generation cohort = 45</td><td>USA</td><td>Cross-sectional (multi-generational)</td><td>Extensive clinical evaluations and bioassays, original focus on cardiovascular diseases</td><td>MRI (T2w)</td><td>NA</td><td>Genotyping, WGS</td><td>Methylation</td><td></td><td>Transcriptomics, metabolomics</td><td>https://www.wachi.alnl.nih.gov/projects/gap/cgi-bin/study/cgi-study.id=phu000724.v9.p13</td><td>[209, 210]</td><td>Data collected over two generations of individuals; cardiovascular MRI A subset of individuals are used to identify rare variants influencing brain imaging phenotypes. Data included in NHLBI TOPMed</td></tr></tbody></table>
<table><thead><tr><th>Women's Health Initiative</th><th>2129</th><th>50-79</th><th>USA</th><th>Cross-sectional</th><th>Clinical evaluations and bioassays; cardiovascular diseases, women only</th><th>NA</th><th>NA</th><th>Genotyping</th><th>Methylation</th><th>NA</th><th>Transcriptomics, metabolomics, miRNA</th><th>https://www.wadi.nlm.nih.gov/projects/gap/cgibin/study/cgistudy_id=phs001335.v2.p3</th><th>[211]</th><th>Women only</th></tr></thead><tbody><tr><td>Sporadic ALS<br>Australia<br>Systems<br>Genomics<br>Consortium<br>(SALSA-SGC)</td><td>1395</td><td>Predicted mean age (from DNA methylation, 60.4 (controls), 62.9 (cases)</td><td>Australia</td><td>Cross-sectional</td><td>Cae control of amyotrophic lateral sclerosis</td><td>NA</td><td>NA</td><td>Genotyping</td><td>Methylation</td><td>NA</td><td>NA</td><td>https://www.wadi.nlm.nih.gov/projects/gap/cgibin/study/cgistudy_id=phs002068.v1.p1</td><td>[212]</td><td></td></tr><tr><td>System Genomics of Parkinson's Disease (SGPD)</td><td>2333</td><td>Predicted mean age (from DNA methylation), mean 70.05, range 26-93</td><td>Australia and New Zealand</td><td>Cross-sectional</td><td>Cae control of Parkinson's disease</td><td>NA</td><td>NA</td><td>Genotyping</td><td>Methylation</td><td>NA</td><td>NA</td><td>https://www.wadi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE145361</td><td>[213]</td><td></td></tr><tr><td>Genetics of DNA Methylation Consortium (nDMC)</td><td>32,851</td><td>0-91</td><td>Worldwide</td><td>Cross-sectional</td><td>Multiple diseases, but also healthy aging cohorts</td><td>NA</td><td>NA</td><td>Genotyping</td><td>Methylation</td><td>NA</td><td>NA</td><td>http://www.godmc.org.uk/coborts.html</td><td>[79]</td><td>Consortium gathering 38 independent studies, data access to be obtained from each sub-sample</td></tr><tr><td>Pediatric Genetics Consortium (FGC)</td><td>By 2025, -2.5 million cases of psychiatric disorders</td><td>All ages</td><td>Worldwide</td><td>Cross-sectional</td><td>Psychiatric disorders, substance use disorders, and neurology: Major depressive disorder, cannabis use disorder, alcohol use disorder, schizophrenia, anorexia, bipolar, ADHD, Alzheimer's</td><td>NA</td><td>NA</td><td>Genotyping and sequencing</td><td>Expression and methylation data available in some working groups</td><td>NA</td><td>NA</td><td>https://www.wadnc.edu/Pge/share methods/open-source-philsophyry/</td><td>[214, 215]</td><td>Consortium organized around disease/ trait working groups</td></tr><tr><td>The genetic epidemiology of asthma in Costa Rica</td><td>4347</td><td>6-12</td><td>Costa Rica</td><td>Cross-sectional</td><td>Asthma cases and controls</td><td>NA</td><td>NA</td><td>WGS</td><td>NA</td><td>NA</td><td>NA</td><td>https://www.wadi.nlm.nih.gov/bioproject/?term=PRINA295246</td><td>[216]</td><td>A subset of individuals from this sample/ accession is used in Pharmacogenetic drug response in racially diverse children with asthma. Data included in NHLBI TOPMed</td></tr><tr><td>Coronary Artery Risk Development in Tonic Adults (CARDIA)</td><td>3425</td><td>18-30</td><td>USA</td><td>Longitudinal</td><td>Coronary Artery Risk</td><td>NA</td><td>NA</td><td>WGS</td><td>NA</td><td>NA</td><td>NA</td><td>https://anlvproject.org/nctp/data/studies/phs00101612</td><td>[217]</td><td>A subset of longitudinal data of 30% self-identified Black and White participants from the CARDIA study were used to study multi-ethnic polygenic risk score</td></tr></tbody></table>

(continued)