Machine Learning in Risk Prediction

HomeHypertensionVol. 75, No. 5Machine Learning in Risk Prediction Free AccessEditorialPDF/EPUBAboutView PDFView EPUBSections ToolsAdd to favoritesDownload citationsTrack citationsPermissions ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toFree AccessEditorialPDF/EPUBMachine Learning in Risk Prediction Johan Sundström and Thomas B. Schön Johan SundströmJohan Sundström Correspondence to Johan Sundström, Uppsala University, Uppsala, Sweden. Email E-mail Address: [email protected] From the Clinical Epidemiology Unit, Department of Medical Sciences (J.S.), Uppsala University, Sweden George Institute for Global Health, University of New South Wales, Sydney, Australia (J.S.). Search for more papers by this author and Thomas B. SchönThomas B. Schön Division of Systems and Control, Department of Information Technology (T.B.S.), Uppsala University, Sweden Search for more papers by this author Originally published16 Mar 2020https://doi.org/10.1161/HYPERTENSIONAHA.120.13516Hypertension. 2020;75:1165–1166This article is a commentary on the followingValue of a Machine Learning Approach for Predicting Clinical Outcomes in Young Patients With HypertensionOther version(s) of this articleYou are viewing the most recent version of this article. Previous versions: March 16, 2020: Ahead of Print See related article, pp 1271–1278Artificial intelligence is a broad term describing systems that show intelligent behavior by analyzing their context and taking actions to achieve certain goals. Machine learning (ML) is a field that sits at the heart of almost all modern artificial intelligence and data science solutions, and that gives computers the ability to learn without being explicitly programmed for the task at hand.1,2 Recent advances in ML methods, combined with an exponentially increasing availability of digital healthcare data, forebode an avalanche of explorations of ML in medical research. We welcome it and believe that ML methods will be an important complement to traditional statistical methods for clinical as well as basic research. For effective application of the scientific method to find the right place for ML in medicine, profound clinical, epidemiological, and statistical knowledge will be important. Clinical researchers need to learn basics of ML methods, ML researchers need to learn basics of medicine and epidemiology, and they need to collaborate with each other and together steer the exploration of the usefulness of ML to the right questions and settings, using appropriate methods for their evaluation.Situations more suited to ML are those where one is only interested in classification of conditions with a high signal-to-noise ratio and does not want to draw conclusions about individual independent variables. Examples are pattern recognition in digital images or digital ECG tracings. These can often use the actual data from the systems that produced the data, which may facilitate transitions to a clinical application. Databases for ML typically need to be very large; the methods are data hungry. Traditional statistical models typically have firm assumptions of the model generating the observed data, whereas ML methods often lack such assumptions or have more flexible assumptions, which makes them less stable in small data sets. While many clinician researchers have a grasp on the sample sizes needed for traditional statistical modeling, sample sizes needed for ML methods are often 10-fold larger.3Situations more suited to traditional statistical modeling are those where one wants to test hypotheses relating to one independent variable, such as in a clinical trial; in low signal-to-noise ratio situations such as most clinical risk prediction scenarios; or if one wants to specify confidence limits around a prediction, which should be but seldom is the case in clinical care.Since prediction tasks are likely to constitute a large portion of the proposed applications of ML in medicine, knowledge on how to evaluate prediction models is probably a good starting place. Confusion on how to develop and evaluate prediction models is widespread.4 Knowledge about measures of discrimination and calibration will be important, as will knowledge of limitations of measures relying on sensitivity and specificity,5 of the clinically important ranges of absolute risk in the target population, relevant time horizons, needs to recalibrate prediction models developed in one scenario if tested in another scenario,6 and of costs of misclassification, both false negatives and false positives.An example of what is likely to come is the article by Wu et al7 in the present issue of the journal. They compared an ML-derived prediction model to the Framingham risk score and a model derived using traditional methods in 508 young persons with hypertension, among which 42 outcome events occurred. It follows the known patterns of comparisons with the Framingham risk score,4 and previous comparisons of ML models with traditional statistical prediction models, but with a 600-fold lower number of events.8 No differences in performance could be noted, as expected with such suboptimal circumstances for both development and evaluation of any prediction model. Of note, also in optimal circumstances, similar performance of ML-derived and established risk scores has been observed.8Diving deeper into ML, the 4 cornerstones of ML are (1) data, (2) mathematical model, (3) learning algorithm, and (4) decision/control. Modern ML solutions (such as deep learning) typically require very large data sets. The second cornerstone is the mathematical model, which amounts to a compact representation of the data that in a precise mathematical form captures the key properties of the underlying situation. An important lesson of contemporary ML is that the best predictive performance is often obtained from flexible models. The most popular flexible model is currently provided by the deep neural networks underpinning all of deep learning.9 Another very promising way of building flexible models is provided by the gaussian process,1 which is significantly less data hungry compared with the deep neural networks. The learning algorithm is used to compute the unknown variables inherent in the model by making use of the data. This is where the measured data get turned into useful information in the form of some kind of hidden patterns. The fourth and final cornerstone consists in algorithms that automatically make decisions to influence the surrounding reality in one way or another.Wu et al have employed the most commonly used form of ML, namely supervised learning, where inputs and the corresponding labeled outputs are used as training data. With these training data, the learning algorithm computes a model that it then used together with new unseen data to make predictions. The hope is the same as in most ML systems, namely that with enough training data the computer can make good predictions from previously unseen data, that is, it can generalize to new input data that were not used during the training.The mathematical model used by Wu et al consists of a sequence of simple - or weak - decision trees that are trained using modified versions of the original training data. A weak classifier is only slightly better than random guessing. The idea underlying the boosting algorithm employed is that even a weak classification model can capture important aspects of the data. Boosting makes use of a sequential construction of classifiers, where the idea is that each model tries to correct for the mistakes made by the previous one. In the process, the training data are modified at each iteration so that the subsequent models focus on explaining the data for which the model is currently performing poorly. The resulting model is then a combination of all weak classifiers. Wu et al made use of XGBoost, which is a highly popular instance of the boosting idea, where gradient information is used in training the models, akin to what is done in most numerical optimization algorithms. In the current study, the authors make use of a nonstandard combination of recursive feature elimination and XGBoost. The idea is that recursive feature elimination will help in pinpointing clinically relevant explanatory variables.In sum, ML has great potential to complement traditional statistical modeling in medicine, but the development and evaluation of ML methods still needs to be done in the framework of proper clinical epidemiology. We propose that the development is focused on the most promising clinical scenarios, using adequately large data sets, and that models are evaluated using proper methods. Once promising models have been developed and externally validated, interesting questions around implementation in clinical care will ensue, involving transparency and responsibility. Fears of black-box solutions and risks of automation bias10 can perhaps be mitigated by highlighting variable importance11,12 to increase transparency and by providing CIs around predictions. Eventually, the future of ML-based clinical decision support tools will depend on them being evaluated in the same way as all medical innovations intended for clinical use, as part of evidence-based medicine.Sources of FundingThis study was supported by the Uppsala University, the Kjell and Märta Beijer Foundation, Anders Wiklöf, and the Wallenberg AI, Autonomous Systems and Software Program funded by Knut and Alice Wallenberg Foundation.DisclosuresNone.FootnotesThe opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.Correspondence to Johan Sundström, Uppsala University, Uppsala, Sweden. Email johan.[email protected]uu.seReferences1. Ghahramani Z. Probabilistic machine learning and artificial intelligence.Nature. 2015; 521:452–459. doi: 10.1038/nature14541CrossrefMedlineGoogle Scholar2. Jordan MI, Mitchell TM. Machine learning: trends, perspectives, and prospects.Science. 2015; 349:255–260. doi: 10.1126/science.aaa8415CrossrefMedlineGoogle Scholar3. van der Ploeg T, Austin PC, Steyerberg EW. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.BMC Med Res Methodol. 2014; 14:137. doi: 10.1186/1471-2288-14-137CrossrefMedlineGoogle Scholar4. Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the framingham risk score.JAMA. 2009; 302:2345–2352. doi: 10.1001/jama.2009.1757CrossrefMedlineGoogle Scholar5. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction.Circulation. 2007; 115:928–935. doi: 10.1161/CIRCULATIONAHA.106.672402LinkGoogle Scholar6. D’Agostino RB, Grundy S, Sullivan LM, Wilson P; CHD Risk Prediction Group. Validation of the framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.JAMA. 2001; 286:180–187. doi: 10.1001/jama.286.2.180CrossrefMedlineGoogle Scholar7. Wu X, et al. Value of a machine learning approach for predicting clinical outcomes in young patients with hypertension.Hypertension. 2020; 75:1271–1278. doi: 10.1161/HYPERTENSIONAHA.119.13404LinkGoogle Scholar8. Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N. Can machine-learning improve cardiovascular risk prediction using routine clinical data?PLoS One. 2017; 12:e0174944. doi: 10.1371/journal.pone.0174944CrossrefMedlineGoogle Scholar9. LeCun Y, Bengio Y, Hinton G. Deep learning.Nature. 2015; 521:436–444. doi: 10.1038/nature14539CrossrefMedlineGoogle Scholar10. Lyell D, Coiera E. Automation bias and verification complexity: a systematic review.J Am Med Inform Assoc. 2017; 24:423–431. doi: 10.1093/jamia/ocw105CrossrefMedlineGoogle Scholar11. Tison GH, Zhang J, Delling FN, Deo RC. Automated and interpretable patient ECG profiles for disease detection, tracking, and discovery.Circ Cardiovasc Qual Outcomes. 2019; 12:e005289. doi: 10.1161/CIRCOUTCOMES.118.005289LinkGoogle Scholar12. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, Liu PJ, Liu X, Marcus J, Sun M, et al. Scalable and accurate deep learning with electronic health records.NPJ Digit Med. 2018; 1:18. doi: 10.1038/s41746-018-0029-1CrossrefMedlineGoogle Scholar Previous Back to top Next FiguresReferencesRelatedDetailsCited By Thong E, Ghelani D, Manoleehakul P, Yesmin A, Slater K, Taylor R, Collins C, Hutchesson M, Lim S, Teede H, Harrison C, Moran L and Enticott J (2022) Optimising Cardiometabolic Risk Factors in Pregnancy: A Review of Risk Prediction Models Targeting Gestational Diabetes and Hypertensive Disorders, Journal of Cardiovascular Development and Disease, 10.3390/jcdd9020055, 9:2, (55) Related articlesValue of a Machine Learning Approach for Predicting Clinical Outcomes in Young Patients With HypertensionXueyi Wu, et al. Hypertension. 2020;75:1271-1278 May 2020Vol 75, Issue 5 Advertisement Article InformationMetrics © 2020 American Heart Association, Inc.https://doi.org/10.1161/HYPERTENSIONAHA.120.13516PMID: 32172625 Originally publishedMarch 16, 2020 PDF download Advertisement SubjectsEpidemiologyMachine Learning and Artificial Intelligence