Reporting on deep learning algorithms in health care (2019)

Presentation of statistical metrics has been described previously in clinical research, epidemiology, and machine learning; however, in-depth discussion for the field of deep learning is scarce. Deep learning using convolutional neural networks has gained substantial traction in medicine; however, inconsistencies exist in the literature on how predicted results from deep learning are reported. Given the rapid pace of deep-learning research in the medical field, the over-reliance on specific statistical methods (eg, mean absolute error), in recent health-care-related deep learning papers1Poplin R Varadarajan AV Blumer K et al.Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning.Nat Biomed Eng. 2018; 2: 158-164Crossref PubMed Scopus (742) Google Scholar, 2Ardila D Kiraly AP Bharadwaj S et al.End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.Nat Med. 2019; 25: 954-961Crossref PubMed Scopus (720) Google Scholar, 3Bellemo V Lim ZW Lim G et al.Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.Lancet Digital Health. 2019; 1: e35-e44Summary Full Text Full Text PDF PubMed Scopus (121) Google Scholar, 4Ehteshami Bejnordi B Veta M Johannes van Diest P et al.Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA. 2017; 318: 2199-2210Crossref PubMed Scopus (1418) Google Scholar, 5Esteva A Kuprel B Novoa RA et al.Dermatologist-level classification of skin cancer with deep neural networks.Nature. 2017; 542: 115-118Crossref PubMed Scopus (54) Google Scholar, 6Ting DSW Cheung CY Lim G et al.Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes.JAMA. 2017; 318: 2211-2223Crossref PubMed Scopus (999) Google Scholar might not provide complete information. For example, in the case of using mean absolute error for evaluation, proportional bias cannot be detected, which is important when evaluating the agreement between predicted results from deep learning and ground truth measurement (appendix p 1). In this instance, additional methods, discussed in this Comment, should be considered to allow more comprehensive analysis of the results. We hereby discuss considerations and recommendations that might be useful to better assess the quality and potential utility of deep learning algorithms for both continuous (eg, blood pressure and weight) and binary (eg, disease vs no disease) health outcomes. Mean absolute error with or without scatter plots is frequently presented for comparisons between the actual ground truth values of continuous health parameters (eg, blood pressure and weight) and the deep learning predicted measurements. Mean absolute error takes the mean of absolute differences between deep learning predicted and ground truth values. This approach potentially masks other useful information—ie, mean absolute error does not provide a clear indication about whether there is an overestimation or underestimation, and by how much. For instance, in appendix p 1, the scatter plot shows the line of best fit obtained by linear regression and seems to suggest that the deep learning algorithm provides good prediction for blood pressure. However, important information such as systemic and proportional error (ie, when the difference between the two measurements is proportional to the magnitude of measurements) could not be determined from this scatter plot. By contrast, Bland-Altman plots allow evaluation of fixed and proportional biases, together with presentation of limits of agreement (the confidence limits of the difference between the deep learning-predicted and the ground truth measurements). Therefore, the Bland-Altman plot could be a useful complementary statistical and graphical method in addition to typical correlation analysis. However, mean absolute error might also be complemented with root mean square error (RMSE), which is more sensitive to outliers. To illustrate this point, we evaluated the correlation between ground truth and values predicted by deep learning for diastolic blood pressure in two scenarios: with and without obvious outliers (appendix p 2). The mean absolute error values are largely similar in both scenarios (7·54 mm Hg without outliers and 7·46 mm Hg with obvious outliers). However, the values are more noticeably different when evaluating using RMSE (9·58 mm Hg without outliers and 10·30 mm Hg with obvious outliers). This finding suggests that RMSE might be a useful method and complementary to mean absolute error. In binary classification (eg, disease vs no disease), the area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity are commonly presented (appendix p 4).2Ardila D Kiraly AP Bharadwaj S et al.End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.Nat Med. 2019; 25: 954-961Crossref PubMed Scopus (720) Google Scholar, 3Bellemo V Lim ZW Lim G et al.Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study.Lancet Digital Health. 2019; 1: e35-e44Summary Full Text Full Text PDF PubMed Scopus (121) Google Scholar, 4Ehteshami Bejnordi B Veta M Johannes van Diest P et al.Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA. 2017; 318: 2199-2210Crossref PubMed Scopus (1418) Google Scholar, 5Esteva A Kuprel B Novoa RA et al.Dermatologist-level classification of skin cancer with deep neural networks.Nature. 2017; 542: 115-118Crossref PubMed Scopus (54) Google Scholar, 6Ting DSW Cheung CY Lim G et al.Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes.JAMA. 2017; 318: 2211-2223Crossref PubMed Scopus (999) Google Scholar, 7Hannun AY Rajpurkar P Haghpanahi M et al.Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network.Nat Med. 2019; 25: 65-69Crossref PubMed Scopus (1115) Google Scholar, 8Li X Zhang S Zhang Q et al.Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study.Lancet Oncol. 2019; 20: 193-201Summary Full Text Full Text PDF PubMed Scopus (187) Google Scholar The receiver operating characteristic curve shows sensitivity and 1–specificity values for all possible cutoffs. AUROC is commonly used to summarise the general performance of a classification algorithm. However, few studies evaluate accuracy (ie, total number of correct classifications divided by total number of cases), positive predictive value, or negative predictive value when reporting classification-related results. Although AUROC provides a general evaluation of the deep learning-based classification performance, for further clinical implementation, a specific classification threshold must be chosen. By altering the classification threshold, the sensitivity, specificity, positive predictive value, and negative predictive value will change accordingly. The classification threshold is commonly selected to maximise informedness, as measured by Youden's index, which is a balanced maximisation of sensitivity and specificity.9Youden WJ Index for rating diagnostic tests.Cancer. 1950; 3: 32-35Crossref PubMed Scopus (7139) Google Scholar When presented with imbalanced datasets, evaluation of algorithm performance based solely on AUROC, sensitivity, and specificity has limitations and should be interpreted with care. When we evaluated a dataset with a small proportion of disease cases (5%), the deep learning algorithms showed an AUROC of 0·957, which appears to suggest excellent classification performance with relatively high specificity of 89·8% and sensitivity of 98·6% (appendix p 3). However, when the same data were evaluated using the precision–recall curve, which is meant to show the trade-off between recall (ie, sensitivity) and precision (ie, positive predicted value), the area under the precision–recall curve (AUPRC) was only 0·348, suggesting that the deep learning algorithm had poor precision (ie, among cases that were positive according to the deep learning algorithm, the probability of actually having disease was low). This outcome is mainly because the dataset was imbalanced, with substantially more healthy controls than cases. The example in appendix p 3 shows that AUPRC and positive predictive value might provide additional information on precision in imbalanced data. However, when the dataset is dominated by disease cases, negative predictive value might be more informative. Taken together, depending on the nature of the datasets (ie, large or small number of cases) and the purpose of evaluations (ie, in addition to general evaluation of detection performance), AUPRC, positive predictive value, and negative predictive value are useful complementary assessments to conventional evaluations of AUROC, sensitivity, and specificity. In view of the limitations in presenting results of deep learning algorithms based on just one or very few methods, as illustrated earlier, we describe statistical methods that should be considered when evaluating different types of health outcomes (table). Overall, different measures and ways of reporting should be considered depending on the aims, context, and outcome of the study.TableStatistical methods for the evaluation of health outcomesDescription of statistical methodContinuous outcomesFrequently used methodsScatter plot (mean absolute error; R2 or correlation)Suggested complementary methodsBland-Altman plot*Useful for detection of proportional and systematic bias. (limits of agreement; evaluation of systemic bias; evaluation of proportional bias); root mean square error†Useful when outliers (ie, very large differences between deep learning predictions and ground truth) are present.Binary outcomesFrequently used methodsROC and AUROC; accuracy, sensitivity, or specificity‡All useful for evaluation of clinical implementation and determination of cutoffs for disease screening.Suggested complementary methodsPositive and negative predictive value; precision recall curve§Provides additional information on precision for imbalanced data. (area under the precision–recall curve)ROC=receiver operating characteristic curve. AUROC=area under the ROC curve.* Useful for detection of proportional and systematic bias.† Useful when outliers (ie, very large differences between deep learning predictions and ground truth) are present.‡ All useful for evaluation of clinical implementation and determination of cutoffs for disease screening.§ Provides additional information on precision for imbalanced data. Open table in a new tab ROC=receiver operating characteristic curve. AUROC=area under the ROC curve. THR was a scientific advisor to a start-up company, Medi Whale, and received stock as a part of the standard compensation package. TYW and DSWT are co-inventors of a patent on the various deep learning systems in ophthalmology; potential conflicts of interests are managed according to institutional policies of the Singapore Health System (SingHealth) and the National University of Singapore. All other authors declare no competing interests. Download .pdf (.29 MB) Help with pdf files Supplementary appendix