A novel approach to maximizing data fidelity in clinical microsurgery research

Plastic surgery research is commonly based on data obtained from institutional databases. Traditionally, these databases, whether retrospective or prospective, derive their information from the electronic medical record (EMR) and rely on two basic processes: data entry (into EMR), followed by data retrieval and reentry (from EMR for transfer to database). This data is generally assumed to be accurate, consistent, and complete, although this has not been previously verified empirically. However, both processes inherently are potentially susceptible to error. In addition, factors such as the variable type or personnel type involved may impact the quality of data, and therefore the quality of research, drawn from these databases. In this article, we present our experience with databases and data fidelity in clinical microsurgery research. We first sought to investigate the quality of data, including sources of error, within a model of a traditional institutional database. Taking these sources of variation into account, we developed an approach to database management that can maximize data fidelity without the need for additional resources. After institutional review board approval, a retrospective case series study was conducted. A model of a standard institutional plastic surgery database was devised involving data entry and data retrieval/ reentry steps. A random sample of 25 consecutive patients who had previously undergone abdominally based microsurgical breast reconstruction by any of five microsurgeons was chosen. A set of 10 data points commonly used in research was collected per patient (e.g., ischemia time, complications). Chart review was performed by four categories of personnel: attending surgeon (1), fellow/senior residents (2), junior residents (2), and medical students (2). Data collected by the attending surgeon served as the standard for completeness of the EMR and the dataset against which data collected by the other raters was compared for accuracy. To assess accuracy, percent agreement was calculated for all variables. To account for random effects in accuracy, Cohen's kappa statistic was calculated for nominal and free answer variables, and intraclass correlation coefficient (ICC) for continuous variables (Cohen, 1960). For consistency between multiple raters, Fleiss's kappa was calculated for nominal and free answer variables, and ICC for continuous variables (Fleiss, 1971). Analysis was performed using SPSS version 25 (IBM Corp, Armonk, NY). There were 13 unilateral and 12 bilateral cases, for a total of 346 data points collected per rater (2,422 total data points). Of the 346 data points, 24 (7%) were missing from the EMR. With respect to accuracy, overall percent agreement was 85% (range, 71-99%). When accounting for random effects, the lowest accuracy was demonstrated for perforator row, 30-day complications, flap type, and prior radiation, all of which demonstrated a Cohen's kappa <0.80, falling below the threshold for acceptable accuracy (Landis & Koch, 1977; McHugh, 2012). There were no clear trends between level of training of personnel. With respect to consistency, the lowest degree of reliability was observed with flap type, recipient artery, 30-day complications, and perforator row, which demonstrated Fleiss' kappa values in the range of 0.60to 0.79, indicating reliability of only 35-63% (McHugh, 2012). Due to the observed limitations in data fidelity associated with a traditional institutional database, we sought to devise an improved system of database management that specifically takes sources of data variation into account. Compliance was also considered; that is, attending surgeons actually using the database and not being deterred by inconvenience or excessive effort involved. This database was designed using Research Electronic Data Capture (REDCap), a free, publicly available, and Health Insurance Portability and Accountability Act (HIPAA)-compliant platform, originally designed to support data capture for research studies (REDCap, https://projectredcap.org/). REDCap entails a web-based interface where data entry can be performed from any computer independent of local data or program files, and has functionality to import data fields into prose form for documentation, as well as searchability for data mining. In the proposed model, two actions are required to record a patient encounter (such as a surgical procedure or clinic visit), which simultaneously achieve both database entry and required EMR documentation: