Title : A case study evaluating the portability of an executable computable phenotype algorithm across multiple institutions and EHR environments Corresponding

Electronic health record (EHR) algorithms for defining patient cohorts are commonly shared as free-text descriptions that require human intervention to both interpret and implement. We developed the Phenotype Execution and Modeling Architecture (PhEMA, http://projectphema.org) to author and execute standardized computable phenotype algorithms. With PhEMA, we converted an algorithm for benign prostatic hyperplasia, developed for the electronic Medical Records and Genomics network (eMERGE), into a standards-based computable format. Eight sites (7 within eMERGE) received the computable algorithm and 6 successfully executed it against local data warehouses and/or i2b2 instances. Blinded random chart review of cases selected by the computable algorithm shows PPV ≥90%, and 3 out of 5 sites had >90% overlap of selected cases when comparing the computable algorithm to their original eMERGE implementation. This case study demonstrates potential use of PhEMA computable representations to automate phenotyping across different EHR systems, but also highlights some ongoing challenges. INTRODUCTION Electronic health records (EHRs) are designed primarily for clinical care and operations, which introduces limitations to the data when used for secondary purposes. These include data availability, accuracy, and information only available in unstructured narrative text. [1-3] To navigate these issues, researchers often develop “phenotype algorithms”– a set of criteria that specifies the analysis of data for identifying and studying patient populations with a given biomedical condition or indication. [4, 5] When replication of results or additional statistical power is needed, a phenotype algorithm may be shared with other institutions for execution on their EHR data, using various methods and tools[6]. A common method used to share algorithms is as free-text descriptions of the algorithmic logic, possibly augmented with flowcharts and lists of codes from medical vocabularies. Implementing this description in a computable form (such as a database query) requires humans to interpret and make decisions for deployment, which can be error-prone and time-consuming. [7] An approach pursued via common data models (CDMs) is to organize EHR data according to a common standard. A CDM, such as those used by Observational Health Data Sciences and Informatics (OHDSI),[8] the National Patient-Centered Clinical Research Network (PCORnet),[9] and Informatics for Integrating Biology & the Bedside’s (i2b2’s)[10] Shared Health Research Informatics NEtwork (SHRINE),[11] can enable cross-site queries since the semantic and syntactic data organization are shared, but require transforming data to the CDM. Within the electronic Medical Records and Genomics (eMERGE) network,[12] the Phenotype KnowledgeBase (PheKB)[13] was created as an online environment to support collaboratively developing, validating, and sharing electronic phenotype algorithms across institutions, with a forum for clarifying implementation details as is often necessary. Truly portable phenotypes requiring little or no human interpretation and data transformation, would facilitate increasing the number of implementing institutions, but are not widely available. To address this gap, we have developed the Phenotype Execution and Modeling Architecture (PhEMA; http://projectphema.org),[14] an open-source infrastructure for standards-based authoring, sharing and execution of phenotyping algorithms. PhEMA uses the National Quality Forum’s (NQF’s) Quality Data Model (QDM) and HL7’s Health Quality Measure Format (HQMF)[15] to unambiguously model phenotype definitions, and enable the execution of these phenotype algorithms against different data representations, including local data warehouses (LDWs), and i2b2 data repositories. We describe a realworld use case for PhEMA: a computable algorithm for identifying patients with benign prostatic hyperplasia (BPH) from EHRs. METHODS BPH Phenotype Algorithm BPH was chosen as a test phenotype for PhEMA in part because the case identification algorithm contains four of the most commonly used EHR data elements (demographics, diagnoses, procedures, and medications), all of the Boolean logical operators (And, Or, Not), a temporal relationship (age), and one aggregation function (count). Since the majority of sites in this study recently executed the algorithm for eMERGE, it also provided a baseline for comparisons against the results using PhEMA. A single institution (Vanderbilt University) defined the original BPH phenotype algorithm for the eMERGE study (referred to as the “original algorithm”), which was modified by the authors for implementation within PhEMA (referred to as the “PhEMA algorithm”). The PhEMA algorithm used a combination of standard medical terminology codes for medications (RxNorm and NDC), diagnostic codes (ICD-9), and procedure codes (CPT). Although the original algorithm used natural language processing (NLP) on patients’ problem lists, NLP was not included in the PhEMA algorithm due to a lack of standard representations of NLP. [16] The study population was defined as males age 40 and older who had no evidence in their EHR of prostate, penile, urethral, or bladder cancer. Within the PhEMA algorithm, the exclusion used just ICD-9 billing codes, although the original algorithm also used ICD-O-3 codes from tumor registries and mentions of these cancers in the problem list. The PhEMA algorithm selected cases only, who were defined as those in the study population with any BPH-related ICD-9 codes on 2 separate days, plus at least one BPH medication or BPH-related surgery code (Figure 1). Translation of Algorithm into a Standardized, Executable Format The modified PhEMA algorithm to detect BPH cases was represented using the PhEMA Authoring Tool (PhAT) (available at: https://github.com/PheMA/bph-use-case), by one of the authors (LVR). Since PhEMA relies on the presence of value sets (groups of medical terminology codes) to represent different medical concepts (e.g., medications, diagnoses), we first looked at the Value Set Authority Center (VSAC)[17] for existing value sets. Outside of the value set of “Male”, no existing value sets met the need of our algorithm; therefore, the authors defined and published the necessary value sets within the VSAC for general use (also available at: https://github.com/PheMA/bph-use-case). Given the use of QDM 4.1 [16, 18] within PhEMA, which accounts for the status of a data element (e.g., an active diagnosis, as opposed to the more abstract “diagnosis”), we included in our definition the relevant statuses for diagnoses, medications and procedures. This approach allows a more precise algorithm definition, as it removes the ambiguity of which statuses are appropriate. The phenotype definition was exported from the PhAT into two executable KNIME workflows (KNIME AG, Zurich Switzerland, Figure 2): one workflow executed query definitions using i2b2 messaging (Figure 2A), and the other executed against an LDW (Figure 2B). KNIME was chosen as the execution engine as it is freely available, runs on multiple operating systems, and, most importantly, can connect directly to external systems that expose a web API. Furthermore, KNIME uses a modular graphical workflow interface that allows encapsulation of algorithm logic (the same across sites) from configuration details (variable between sites), allowing each site to configure the connection to their data without having to edit the algorithm’s logic. [18, 19] Upon review of the exported KNIME workflows, we identified opportunities to optimize their execution before distributing to sites. First, within sites’ i2b2 instances, diagnosis status (active, resolved or inactive) was not explicitly described; thus, the 3 queries (one for each status) were collapsed into a single query. Second, by default the workflow returned multiple types of results within i2b2 for each query – a count, a list of patients, and a list of events. We edited the workflow to only return patient sets, to further reduce execution time. Finally, we removed the temporal relationship between BPH diagnoses and instead required patients have ≥2 BPH diagnoses, as some sites did not have visits associated with diagnoses in their i2b2 data. Once given to the implementation site, each workflow required up to two local customization steps. First, the site must specify the connection details for their repository (i2b2 or LDW). Second, the site performed data customization if needed. For i2b2, this involved updating the ontology mapping so the correct concepts could be found; and for the LDW, editing template data queries to return the required data elements. Two members of the PhEMA team (JAP, LVR) guided this customization via remote