Developing an ML pipeline for asthma and COPD: The case of a Dutch primary care service

International Journal of Intelligent SystemsVolume 36, Issue 11 p. 6763-6790 RESEARCH ARTICLEOpen Access Developing an ML pipeline for asthma and COPD: The case of a Dutch primary care service Stefano Mariani, Corresponding Author Stefano Mariani stefano.mariani@unimore.it orcid.org/0000-0001-8921-8150 Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Reggio Emilia, Italy Correspondence Stefano Mariani, Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Via Giovanni Amendola, 2, 42122 Reggio Emilia RE, Italy. Email: stefano.mariani@unimore.itSearch for more papers by this authorEsther Metting, Esther Metting orcid.org/0000-0003-1677-3529 Health Technology Assessment, Department of Epidemiology, University of Groningen, University Medical Center, Groningen, The NetherlandsSearch for more papers by this authorMaarten M. H. Lahr, Maarten M. H. Lahr orcid.org/0000-0001-7265-2612 Health Technology Assessment, Department of Epidemiology, University of Groningen, University Medical Center, Groningen, The NetherlandsSearch for more papers by this authorEloisa Vargiu, Eloisa Vargiu orcid.org/0000-0002-3840-628X EURECAT Technology Centre, Digital Health Unit, Barcelona, SpainSearch for more papers by this authorFranco Zambonelli, Franco Zambonelli orcid.org/0000-0002-6837-8806 Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Reggio Emilia, ItalySearch for more papers by this author Stefano Mariani, Corresponding Author Stefano Mariani stefano.mariani@unimore.it orcid.org/0000-0001-8921-8150 Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Reggio Emilia, Italy Correspondence Stefano Mariani, Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Via Giovanni Amendola, 2, 42122 Reggio Emilia RE, Italy. Email: stefano.mariani@unimore.itSearch for more papers by this authorEsther Metting, Esther Metting orcid.org/0000-0003-1677-3529 Health Technology Assessment, Department of Epidemiology, University of Groningen, University Medical Center, Groningen, The NetherlandsSearch for more papers by this authorMaarten M. H. Lahr, Maarten M. H. Lahr orcid.org/0000-0001-7265-2612 Health Technology Assessment, Department of Epidemiology, University of Groningen, University Medical Center, Groningen, The NetherlandsSearch for more papers by this authorEloisa Vargiu, Eloisa Vargiu orcid.org/0000-0002-3840-628X EURECAT Technology Centre, Digital Health Unit, Barcelona, SpainSearch for more papers by this authorFranco Zambonelli, Franco Zambonelli orcid.org/0000-0002-6837-8806 Department of Sciences and Methods for Engineering, University of Modena and Reggio Emilia, Reggio Emilia, ItalySearch for more papers by this author First published: 21 July 2021 https://doi.org/10.1002/int.22568 Esther Metting and Maarten M. H. Lahr contributed equally to this work. [Correction added after online publication on 16 September 2021: Reodering of the author has been done.] AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract A complex combination of clinical, demographic and lifestyle parameters determines the correct diagnosis and the most effective treatment for asthma and Chronic Obstructive Pulmonary Disease patients. Artificial Intelligence techniques help clinicians in devising the correct diagnosis and designing the most suitable clinical pathway accordingly, tailored to the specific patient conditions. In the case of machine learning (ML) approaches, availability of real-world patient clinical data to train and evaluate the ML pipeline deputed to assist clinicians in their daily practice is crucial. However, it is common practice to exploit either synthetic data sets or heavily preprocessed collections cleaning and merging different data sources. In this paper, we describe an automated ML pipeline designed for a real-world data set including patients from a Dutch primary care service, and provide a performance comparison of different prediction models for (i) assessing various clinical parameters, (ii) designing interventions, and (iii) defining the diagnosis. 1 INTRODUCTION Asthma and Chronic Obstructive Pulmonary Disease (COPD) are prevalent chronic respiratory diseases. In total over 339 million people worldwide suffer from asthma and over 65 million people have moderate to severe COPD, which is responsible for 5% of all global deaths.1, 2 Both diseases are treated by a combination of triggers avoidance (like, smoking or allergens), healthy lifestyle habits, and usage of inhaled medications. For the diagnosis of asthma and COPD, spirometry is the gold standard.3 The aim of asthma and COPD treatment is to reduce the impact of the disease on daily life. The severity of asthma may fluctuate over time but is not progressively worsening; on the contrary, COPD is a progressive and potentially deadly disease. The possibility of exacerbations is present in both asthma and COPD patients, and can significantly impact the patients' health status. Therefore, prevention of exacerbations is one of the pivotal goals of the treatment. Exacerbations are treated in primary care with oral antibiotics, corticosteroids or with oxygen in case of the most severe episodes. Current clinical practice is mostly based on personal professional skills acquired by training, past experience and intuition, whereas digital support is often limited to rule-based clinical management systems configured to deliver predefined decision criteria depending on pre-established clinical parameters of the patient.4 However, predictive modelling tools based on statistical models or Artificial Intelligence (AI) techniques, such as machine learning (ML) are gaining traction as an innovative way to interpret the growing amount of patient data available, aimed at achieving more patient-centred and customised care plans.5-7 In particular, within respiratory research the biggest area of expansion is the application of ML within imaging.8 However, the diagnosis of respiratory conditions, such as asthma and COPD, relies on much more than image analysis; it involves taking a patient's history and a physical examination together with pulmonary function tests, and possibly imaging (X-rays, CT scans, and bronchoscopy).9 Personalised and patient-based health-risk assessment is a common example of such a predictive digital tool, and is increasingly used by clinicians to support individual decisions. For example, it is used to predict clinical episodes, like, exacerbations, or to personalise clinical pathways to patient conditions.10 Indeed, clinical pathways definition is another example of a clinical task where usage of digital tools driven by AI techniques is steadily growing. However, to build such predictive modelling tools patient data must be available, possibly accurately reflecting the same real-world conditions that they will be operating on, for instance, regarding imbalanced representation of clinical attributes, missing data, or noisy and badly formatted data. Instead, as emphasised recently in the ML community,11 data sets used for advertising the most accurate prediction models are too often woefully out of touch with reality. For instance, because the best ML models are usually measured against large and curated data sets that lack the noise typical of real-world deployments. In this paper we contribute to the development of predictive modelling in the context of diagnosis and treatment of asthma and COPD patients, with particular attention to the real-world setting where the tool would be deployed: First, we describe the ML pipeline set-up for the exploitation of data of asthma and COPD patients coming from a Dutch primary care service, with particular attention to its development aimed at obtaining an automated pipeline easy to deploy and integrate in existing software stacks. Then, we compare the performance of different prediction models for various clinical parameters, automated diagnosis and suggestion of medical interventions. Finally, we summarise the “lessons learnt” during our research for the benefit of researchers and clinicians willing to perform similar tasks. To the best of our knowledge, as summarised in Section 2.3, this is the first research work addressing both asthma and COPD for the comparison of multiple prediction models across different prediction targets, while relying on primary care data. In the remainder of the paper, Section 2 gives context knowledge to health-risk prediction for asthma and COPD, then Section 3 presents the proposed ML pipeline detailing the techniques exploited for data preprocessing, and model training and scoring, whereas evaluation of the resulting prediction models is in Section 4. Section 5 summarises the limitations of our study and the “lessons learnt” during our research, and finally Section 6 provides for final remarks. 2 BACKGROUND AND RELATED WORKS In this section we provide the reader with the clinical and technological background knowledge necessary to fully understand the context, motivations and goals of our research. Accordingly, we provide a brief account on the current state-of-the-art regarding usage of digital tools for clinical pathways definition, there including literature showing evidence of the effectiveness of ML-based decision support systems (DSS) for predicting various asthma and COPD outcomes (in Section 2.2). Finally we overview what are the recent efforts in providing support to clinicians specifically through ML and in the case of asthma and COPD patients, and what are the main differences with our work. 2.1 Digital tools for clinical pathways Research performed on electronic (or, computerised) clinical pathways can be roughly divided into three macroareas. Clinical pathways analysis: Digital tools are created either to build an in silico representation of clinical pathways, enabling simulations and “what-if” analysis, or to define machine-readable specifications of constraints on pathways to be enforced in clinical practice. Both efforts share the goal of monitoring compliance or identifying bottlenecks and devise out reasons for underperformance of current care plans. In Reference [12], for instance, process mining is used to mine electronic health records and build clinical pathways post hoc. Others13 have instead suggested an ontology-based approach for the definition of pathways in a workflow style. Clinical pathways synthesis: The focus is on an automated definition of clinical pathways, or enforcement and execution thereof, with the goal of practically assisting clinicians in the definition and execution of care plans. Both References [14,15], for instance, propose a semantic rule-engine configured with domain expert knowledge and a suitable rule-set to suggest adaptations to care plans depending on the specific patient conditions or unexpected events. DSS within clinical pathways: Here, emphasis is given to DSS that support both the pathways definition and their execution, but in specific tasks—as in our case. The work in Reference [16], for instance, proposes a recommender system to suggest to clinicians the next steps of disease management depending on the evolving patients conditions. In the following sections, we overview the issue of predicting clinical variables for asthma/COPD patients and the AI techniques that are increasingly used to support such a task, with a focus on ML approaches. 2.2 Health-risk prediction for asthma and COPD Due to the large variation of characteristics in asthma and COPD patients, it can be extremely difficult even for experienced physicians to determine the most effective treatment for each patient. In case of asthma, besides spirometry, the change in Forced Expiratory Volume in 1 second (FEV1) before and after administration of a large inhaled dosage of bronchodilator medications can confirm the diagnosis. However, the absence of such a change does not necessarily mean the absence of asthma, as many patients have normal lung function during diagnostic assessment. Hence, physicians often determine their diagnosis based on the clinical evaluation of symptoms, or via histamine provocation tests.17 In case of COPD, the diagnosis is performed based on patient history, including packyears and the presence of a fixed obstruction. An obstruction is determined using the FEV1 and Forced Vital Capacity (FVC). The severity of COPD is based on lung capacity measured with the proportion of predicted FEV1. However, the burden that patients suffer in daily life activities is only partly determined by the level of obstruction. In addition, symptom questionnaires, such as the Clinical COPD Questionnaire (CCQ) or the COPD Assessment Test (CAT) provides a more comprehensive assessment of the level of disease severity, which is divided into GOLD stages A (least severe), B, C and D (most severe). The optimal treatment is different for each patient and is based on disease severity, symptoms, patient characteristics and individual risk factors. Personalised treatment has proven to be effective in improving patient outcomes, however, it is unlikely that physicians during their limited consultation times can accurately and comprehensively evaluate all of the many aspects that might affect worsening of the disease or triggering of an exacerbation. The human brain is simply not capable of recognising large amounts of longitudinal patterns and interactions between different predictors, as computers can.18 For these reasons, AI can be promising especially when based on real-life large databases and when using ML techniques, for instance, by supporting healthcare providers in predicting the effect of specific treatment approaches.19 In this way AI can be a valuable aid for healthcare professionals and reduce the risk of treatment side effects of patients.21 The use of automatic clinical DSSs may improve the diagnosis and ongoing management of chronic diseases, which currently requires periodic visits to multiple health professionals, disease and medication monitoring, and modification of patient behaviour. The systematic literature review by Fathima et al.20 brings evidence of effectiveness of clinical DSSs in the care of people with asthma. However, they did not found a clear evidence on using them for COPD. Another systematic review of clinical DSSs was performed by Roshanov et al.,22 with the objective to determine if clinical DSSs improve the process of chronic care (in diagnosis, treatment and monitoring) and associated patient outcomes. The authors identified 55 trials that measured and reported the impact of the clinical DSS on the process of care, and/or patient outcome. Out of the clinical DSSs that measured the impact on the process of care, 52% demonstrated a statistically significant improvement, and out of the trials that measured patient outcome 31% demonstrated benefits. Along the same line, Velickovski et al.23 propose a clinical DSS in charge of delivering recommendations. Results show a high degree of accuracy to support COPD case-finding. Moreover, they demonstrate the integration into healthcare providers' workflow through the use of a modular design and service-oriented architecture that connects to existing health information systems already in use. However, most of the DSSs considered in the aforementioned reviews do not focus on the prediction of risk and suggestion of interventions after baseline assessment, as we do, and do not focus on interoperability and portability of the models embedded in the DSS, as we do. The ML pipeline we describe in Section 3 and evaluate in Section 4 is not a DSS per se but as a fully automated ML pipeline implemented in Python can be easily embedded into one. Indeed, such a pipeline could be easily served via a DSS, such as the one described in Reference [24]. 2.3 ML for asthma/COPD The goal of best predicting clinical variables related to asthma/COPD with data-driven approaches is shared by many research works in state-of-the-art literature, and rightfully so, as the benefits that AI techniques may bring to the whole asthma/COPD management are widely recognised.25 However, most of the existing literature differ from our contribution in several aspects, such as data provenance and preprocessing, the approach adopted and outcome of the research. For instance, many works we mention in the following perform heavy preprocessing of data to select the most likely predictors, or get data from specific clinical trials. Also, whereas they want to identify statistically meaningful predictors based on extensive statistical analysis (e.g., uni/multivariate regression methods), we focus on the performance of a fully automated ML pipeline autonomously building a slew of predictive models (also targeting suggestion of interventions) from unfiltered primary care data—that is, data fetched “as is” from a Dutch primary care centre, with no a priori filtering or processing. Nevertheless, we here position our work with respect to with the most similar works to highlight similarities and differences. Most of the related research works perform some form of statistical analysis on data collected at various scales (e.g., primary care vs. controlled trials) to evaluate the predictive value of specific attributes. For instance, in Reference [26] the authors perform univariate analysis to identify likely predictors of COPD exacerbation events, then feed such predictors into a stepwise multivariable logistic regression model, and finally carry out a sensitivity analysis with respect to asthma and smoking pack-years. Hence the first difference with our work is that they preselect a prediction model, whereas we compare a slew within an automated pipeline. They get data from a curated database and further select patients based on eligibility criteria meant to further remove confounding variables (e.g., of patients with other respiratory diseases, such as bronchiectasis). Also, they preselect relevant candidate predictors based on literature and expert knowledge. In contrast, our work relies on primary care data “as is,” and carries out minimal preprocessing automatically—with the purpose of being easily applicable without experts assistance. Finally, their work is aimed at finding the best predictors for COPD exacerbations, whereas our work has a twofold goal: find good prediction models for a few different asthma and COPD-related variables, while also striving to develop a reusable and easy to deploy ML pipeline. For the work in Reference [27] similar considerations can be done, as authors select candidate predictors from literature and through statistical analysis and then build an a priori model to be evaluated using receiving operating characteristic (ROC) area under the curve (AUC), the same scoring metric that we use. They rely on data from selected clinical trials and focus on asthma only. In Reference [28] another statistical retrospective study deals with the issue of determining risk factors associated with asthma and COPD, applying univariate and multivariate logistic regression to determine predictors of future exacerbations. However, they only consider less than 400 patients specifically enrolled in the study. Although our work is considerably different in most aspects, such as sample size, data provenance, models comparison and automation of our ML pipeline, it shares with Reference [28] the inclusion of both asthma and COPD, which complicate the classification task as noted by authors themselves—even if they divided the population into subgroups. Nevertheless, our work provides for a much broader overview about predictive models for asthma and COPD, and it is immediately applicable to primary care data sets since our automated ML can deal with the data preprocessing steps reported in Figure 1 autonomously, as well as training and scoring models automatically as described in Section 3. Figure 1Open in figure viewerPowerPoint The machine learning pipeline. In light grey are the data set dimensions (rows × columns). ACQ, Asthma Control Questionnaire; AUC, Area Under the Curve; CCQ, Clinical COPD Questionnaire; COPD, Chronic Obstructive Pulmonary Disease; kNN, k-Nearest Neighbour; LABA, Long-Acting β-Agonists; LAMA, Long-Acting Muscarinic Antagonists; ROC, Receiving Operating Characteristic; SVC, Support Vector Classification More focussed on automated ML is the recent work started in Reference [29], that is similar to ours in both the goals (providing to clinicians a decision support tool) and the general approach (adopting ML techniques to learn prediction models from data), as authors aim to build a learning healthcare system exploiting an ML pipeline fed with primary care data to learn different asthma attack prediction models. However, they intend to consider asthma only and focus on a single task, that is, prediction of a single clinical variable (asthma attack). This notably simplifies the goal to be achieved with respect to our case, where we want to predict different outcomes, also give advises about treatment plans, and finally consider COPD too—which complicates our goal as it is partially overlapped with asthma symptoms. Obviously, as the work is yet to be concluded and fully reported, we cannot compare results yet. Another recent paper more focussed on ML, and with very similar objectives and methods to ours is, Reference [30] where authors compare a slew of different ML models to predict the risk of readmission for COPD patients. Authors compare logistic regression and its variants, random forests, linear support vector machine (SVM), gradient boosting, multilayer perceptron and also deep learning models, including temporal features, such as convolutional neural networks, recurrent neural networks, long–short term memory and gated recurrent unit. The work is similar to ours in that authors build a mostly automated ML pipeline, with the objective of comparing which ML models could perform best for the target prediction task, which has strong reusability and modularity, hence appear to be easily adapted to slightly different tasks and easily integrated with existing DSSs for usage by clinicians in day-to-day practice—a goal we also have. Furthermore, they have a discussion section which is quite similar to our Section 5, although ours is a bit broader in scope as it covers both technical and nontechnical aspects. However, Reference [30] also differs under several aspects: first of all we do consider asthma too, and multiple prediction targets for the same primary care data set, which notably complicates the learning task, whereas they focus on COPD and a single prediction target; second, they rely on data beyond primary care, and require 12 months of clinical data as inclusion criteria. Nevertheless, even given their more focussed scope, their results are mostly worse than ours, as their best-reported performance has ROC AUC of 0.65—they do not report on precision-recall curves, that may report even worse results as described in Section 3.4. Finally, two more related works are worth mentioning, despite their notable differences with the present paper. Reference [31] is one of the few works we found that considers asthma and COPD altogether. However, the learning task and methods are considerably different from ours: they want to discriminate asthma patients from COPD patients by analysing saliva samples. Furthermore, they rely exclusively on black-box ML models, that may be difficult to introduce in clinical practice due to their low transparency and understandability. However, one interesting aspect of Reference [31] lies in usage of “few shots” (e.g., 0 and 1) learning models, which are able to learn from very few data samples. Reference [32], instead, is interesting for two reasons, even if the scope is quite different from ours, as they deal with the prediction of individual asthma persistence upon clinical input for children under the age of 5 years with an incident asthma diagnosis. First, authors deal with highly imbalanced prediction classes, as for many of our prediction targets, by comparing different undersampling techniques, such as the edited nearest neighbors (ENN) method, that removes instances (of the majority class) whose class label differs from a majority of its k-nearest neighbours (kNN), repeated edited nearest neighbors (R-ENN) that repeats the ENN procedure until the majority of the kNNs for every data point (of the majority class) have the same class label as the data point, and by removing majority class instances of Tomek links, that are pair of instances which are each other's nearest neighbour but are in different classes. Although these techniques may generally improve the prediction outcome, they do so by altering the data set while artificially removing (or, at least, simplifying handling of) the most difficult instances to classify. For these reasons we chose not to exploit such techniques in this paper (as described in Section 3.1), leaving them as future work for comparison with our current results. Second, they use negative predictive value (NPV)-Specificity curves, which are similar to the precision-recall curves we use, as they acknowledge ROC curves are poor for imbalanced classes, too. Given the above analysis, our work is, to the best of our knowledge, the first one to consider both asthma and COPD patients for building an automated prediction pipeline based on primary care data. 3 MATERIAL AND METHODS To both develop the ML pipeline and validate prediction models, we exploited data coming from a Dutch primary care laboratory in the city of Groningen that receives approximately 2000 patients yearly with suspicion of asthma or COPD, that are referred for assessments and treatment advice. Patients receive assessment by a trained laboratory technician according to the American Thoracic Society and European Respiratory Society guidelines, there including respiratory testing with reversibility, medical history, smoking behaviour, Body Mass Index (BMI), medication and inhaler technique evaluation. The primary care physician receives the advice from the pulmonologist directly in his/her electronic patient record. If the pulmonologist advises patients to change the medication regime, then patients are advised to have a follow-up assessment after 3 months to evaluate the effect of the new medication. Instead, if the pulmonologist advises to continue the current treatment policy, the patient is rescheduled for a yearly follow-up. The primary care service shared a data set storing the real-life observational data derived between 2007 and 2017. The data set contains baseline assessment of the clinical conditions of 19 , 077 patients. Attributes describe data, such as age, gender, BMI, family history of the disease, lifestyle habits associated with the disease, such as smoking, spirometry including FEV1, FVC, and reversibility measured by a trained laboratory technician, common symptoms, such as cough, wheeze, and dyspnoea, information about medications including inhalation technique, and symptom questionnaires, such as Asthma Control Questionnaire (ACQ) and CCQ. All mentioned data are assessed by a local pulmonologist through the internet. Diagnosis and treatment advice is sent to the general practitioner (GP) of the patient. Besides baseline, follow-ups are also available at different time points and only for some patients (more on this below). This results in 2454 attributes per patient, made of a set of ≈ 160 attributes repeated over time, for each potential follow-up. Amongst these, the primary care service focusses on: for patient-based health-risk assessment, prediction of clinical variables relevant for diagnosis and prognosis of asthma, COPD and asthma–COPD overlap syndrome (ACOS) patients. In particular, the aim is to build a predictive model for: – the amount of exacerbations at 1 year (0, 1, 2+), predicted after ACQ and CCQ assessment (usually done at baseline assessment), – the ACQ category at 3 and 12 months follow-ups after baseline assessment (controlled, partially controlled and uncontrolled), – the CCQ category at 3 and 12 months follow-ups after baseline assessment (stable, not entirely stable, unstable and very unstable), for clinical pathways definition in the form of suggestions for personalised intervention, the aim is to build a predictive model for: – advising usage of Long-Acting Muscarinic Antagonists (LAMA) after baseline assessment, – advising usage of (low/high dosage of) Inhaled Corticosteroids (ICS), or Long-Acting β-Agonists (LABA), or both after baseline assessment, – advising usage of β 2 bronchodilation after baseline assessment. Also automated diagnosis has been explored, by trying to predict whether the patient has ACOS, COPD or asthma after baseline assessment. Since the data set is extremely sparse and data are not “clean” (as real-life data), many preprocessing steps were necessary before starting with model training. Section 3.1 describes such steps, while Figure 1 depicts the whole ML pipeline described in this section. 3.1 From format conversion to preprocessing The data set has been exported from IBM SPSS propri