B80-3-04 The Utility and Challenges of Extracting Symptom Data From Electronic Health Records Using Natural Language Processing for Cardiovascular Diseases and Obstructive Sleep Apnea

Abstract                                        Rationale                     Clinical notes in the electronic health record (EHR) contain important information beyond ICD codes which can be extracted using natural language processing (NLP). In an on-going World Trade Center (WTC)-related study on cardiovascular diseases (CVD) and obstructive sleep apnea (OSA), we first examined whether CVD and OSA-oriented information can be extracted from EHR by developing and testing the utility of an NLP-based machine learning model.                                                           Methods                     A physician manually reviewed clinical notes of 145 patients in the Mount Sinai Hospital EHR and annotated CVD events (coronary artery disease (CAD), myocardial infarction (MI), stroke, congestive heart failure (CHF)) and OSA status, extracted from sleep study results (in-lab and home-based). A subtype of NLP (bidirectional encoder representations from transformers) model was fine-tuned (n = 43, 80%/10%/10% train/validation/internal test set) with a named-entity recognition task, and tested on 102 subjects not seen by the model (external test set). Relevant ICD codes were obtained. Performance metrics were precision, recall, and F1 score. An F1 score greater than 80% was considered reliable.                                                           Results                     The cohort was male predominant (75.7%) with median age 53 years and median BMI 29.2 kg/m2. Chart review identified 34 patients with CAD, 6 with MI, 5 with CHF, and 4 with stroke. OSA was confirmed in 103/110 patients who had sleep studies done. We identified only 16 patients with CAD-relevant ICD codes and 33 patients with OSA-relevant codes. Due to low prevalence of non-CAD events in our cohort, we restricted classification to CAD and no-CAD. Performance metrics are reported in Table 1.                                          False positives of the NLP model included inability to separate CAD diagnosis from patients’ medical history and family history and inability to distinguish CAD diagnosis from coronary calcium score. False negatives occurred when the model could not locate notes not based in our EHR (but available under Epic-Care Everywhere).                                                           Conclusion                     We highlight the utility and challenges of using NLP to extract accurate clinical CVD data to elucidate relationships with OSA. OSA has high prevalence in this WTC cohort. Clinical documentation for cardiac-related conditions was low as care was provided outside of our system, a common issue in a disintegrated healthcare network. Some patients with CAD or OSA diagnoses did not carry their respective ICD codes in their charts, underlining the importance of extracting clinical information beyond ICD codes. NLP shows promise in searching and returning accurate information for OSA. Additional data is needed to train the model.                     This abstract is funded by: CDC