External Validation of the MEESSI Acute Heart Failure Risk Score

Acute heart failure (AHF) is the most common cause of hospitalization in patients aged 50 years or older, with more than 2 million hospitalizations annually in both Europe and the United States (1, 2). It is a heterogeneous syndrome that is associated with unacceptably high rates of mortality and morbidity, due in part to uncertainty about the intensity of management both in the hospital and immediately after discharge (37). Because most patients with AHF present to the emergency department (ED) for initial treatment, early risk prediction plays a key role in subsequent management (8). In this context, the decision about whether to hospitalize a patient is critically important. However, a mostly subjective and empirical process results in hospitalization of most patients with AHF and large regional differences in the percentage of patients who are discharged directly to home (911). Therefore, a tool for standardized risk stratification of patients with AHF is needed. This would allow high-risk patients to receive prompt and aggressive in-hospital therapy (possibly even in the intensive care unit, with continuous hemodynamic and electrocardiographic monitoring), whereas low-risk patients could be safely discharged to home without exposure to potential hazards, such as nosocomial infections and delirium (16). The MEESSI-AHF (Multiple Estimation of risk based on the Emergency department Spanish Score In patients with AHF) risk score was recently found to provide highly accurate risk stratification of patients presenting to EDs in Spain with AHF (12). It has several advantages over other AHF risk scores, including that it was developed specifically for use in the ED and was not derived from data on hospitalized patients (1316). Given previously reported characteristics of AHF care in Spain, including a much higher rate of outpatient management than in other countries (17), whether the MEESSI-AHF score performs well in other countries is unknown. Thus, our aim was to externally validate the score in unselected patients with AHF who were treated in another European country. Methods Study Design and Population BASEL V (Basics in Acute Shortness of Breath EvaLuation) (ClinicalTrials.gov: NCT01831115) was a prospective, multicenter, diagnostic study that enrolled adult patients presenting with acute dyspnea to the EDs of 2 university hospitals (University Hospital Basel and University Hospital Zurich) and 2 nonacademic tertiary care centers (Kantonsspital Aarau and Kantonsspital Lucerne) in Switzerland. Patients were included regardless of renal function; however, those with terminal kidney failure requiring hemodialysis were excluded. For the current study, only patients with an adjudicated final diagnosis of AHF and a complete set of variables necessary for calculation of the MEESSI-AHF score were included. The study was done according to the principles of the Declaration of Helsinki and was approved by the local ethics committees. Written informed consent was obtained from all patients. The authors designed the study, gathered and analyzed the data according to the STARD (Standards for the Reporting of Diagnostic Accuracy Studies) guidelines for studies of diagnostic accuracy, vouched for the data and analysis, wrote the article, and decided to publish it. Adjudication of Final Diagnosis The final diagnosis of AHF was adjudicated by 2 independent cardiologist-internists who had access to all patients' medical records, including clinical history, physical examination, 12-lead electrocardiograms, laboratory findings, chest radiographs, echocardiograms, lung function test results, computed tomography scans, and response to therapy, as well as autopsy data for patients who died in the hospital. All laboratory findings obtained through the clinician's routine diagnostic work-up were available for this study. These included findings on the natriuretic peptides (B-type natriuretic peptide [BNP] or N-terminal proB-type natriuretic peptide [NT-proBNP]) that current guidelines recommend for diagnosing AHF (class I recommendation) (18, 19). Cases for which there was disagreement about the final diagnosis were reviewed and adjudicated by a third cardiologist. Follow-up Patients were contacted 3, 12, and 24 months after discharge by telephone or in writing. Information about death during follow-up was obtained from hospital medical records, the general practitioner, and the national mortality registry. Calculation of the MEESSI-AHF Risk Score The full model for calculation of the MEESSI-AHF score contains 13 variables collected at patients' admission to the ED: Barthel Index score, systolic blood pressure, age, NT-proBNP level, potassium level, elevated cardiac troponin T (cTnT) level, New York Heart Association class IV disease, respiratory rate, low-output symptoms, oxygen saturation, episode associated with an acute coronary syndrome, left ventricular hypertrophy on electrocardiogram, and creatinine level. In this study, the score was calculated using an established alternative model (model D) that included all of these variables except Barthel Index score, which was not prospectively recorded in our study. All values were the initial ones obtained before treatment was started and were captured by experienced study physicians who were well trained on the study protocol and definition of the study variables in order to reduce interobserver variation. As in the original model, the multivariable risk score (x) for a specific patient was calculated by summing the relevant risk coefficients and the intercept shown in the Table. The probability of death within 30 days was calculated using the logit function [ex/(1 + ex)]. Table. 30-Day Mortality Using a Reduced Multivariable Predictive Model (Model D) for Calculation of the MEESSI-AHF Risk Score* External Validation of the MEESSI-AHF Risk Score The MEESSI-AHF risk score was externally validated using methods described previously (20, 21). In brief, initial validation was done without adjustment to the original prediction rule. It was then recalibrated by simple adjustment of the intercept, which allowed for calculation of a correction factor that was added to the intercept of the original model (20). This correction factor was based on the mean predicted risk from the model on a scale of 0 to 1 (continuous risk prediction) and the observed outcome frequency in the validation cohort. Therefore, it aligned differences in outcome frequencies between the derivation and validation cohorts (see the Appendix). In both methods, the model's discrimination was quantified by the c-statistic and calibration was quantified by the HosmerLemeshow test as well as graphical assessment using a calibration plot. The HosmerLemeshow test statistics were based on the predicted probabilities using the parameter estimates from the original MEESSI-AHF publication. The model's discrimination and calibration were assessed using continuous risk predictions from the model. The slope and intercept of the calibration plot were calculated using the R package rms. Logistic and flexible calibration curves and related statistics were generated using val.prob. ci.2, an established R function (22). Because the sample had fewer than 5000 patients, the smoothing approach selected when calling the val.prob. ci.2 function was loess, which generates a flexible calibration curve and is recommended by the developers of the val.prob. ci.2 function. Patients were classified into the risk groups defined in the original publication (12) according to their predicted probability of death. In addition, we analyzed the number of hospitalized patients in each of the original risk groups. New risk groups were also created according to the risk score distribution in the validation cohort without recalibration. Patients were then divided into quintiles according to their probability of death, with the top quintile divided into 2 deciles. This approach is equivalent to the one used to determine the risk groups in the original model. The cumulative mortality rates over 30 days for the original and new risk groups were plotted using KaplanMeier curves. In a sensitivity analysis, the operating characteristics of the MEESSI-AHF risk score were computed with inclusion of data from patients with missing values. We performed multiple imputation with chained equations to produce 20 imputed data sets for estimation of missing values. All 12 variables used in model D to calculate the MEESSI-AHF score were included in the data imputation model. The score and its c-statistic and HosmerLemeshow P value were calculated in every data set according to the original model. Results were combined by using the Rubin rules (23). Baseline characteristics and candidate predictor variables of the model were compared between patients included in the main analysis and those excluded due to missing values. We also directly compared the diagnostic accuracy of the MEESSI-AHF risk score with that of another validated AHF risk score, the Emergency Heart Failure Mortality Risk Grade (EHMRG) (14), which was designed to predict 7-day mortality. The areas under the curve for 30-day mortality for both models were compared using the method proposed by DeLong and colleagues (24). In addition, we computed the operating characteristics of MEESSI-AHF models that did not include Barthel Index score, cTnT level, or NT-proBNP level in different combinations. Statistical Analysis Continuous variables are presented as medians and interquartile ranges, and categorical variables are expressed as numbers and percentages. Differences in baseline characteristics of the study population were assessed using the MannWhitney U test or the 2 test as appropriate. All hypothesis testing was 2-tailed, and P values less than 0.05 were considered statistically significant. Statistical analysis was done using SPSS for Windows, version 25.0 (IBM), and R, version 3.4. 3 (R Foundation for Statistical Comp