Comparing Patient-Reported Hospital Adverse Events with Medical Record Review: Do Patients Know Something That Hospitals Do Not?

Context Some hospitals review a patient's hospital record to ascertain adverse events during a recent hospital stay. The accuracy of this procedure is unknown. Asking patients about adverse events is another approach. Contribution The authors surveyed randomly chosen patients discharged from all hospitals in Massachusetts. Of 998 patients who returned the surveys and gave permission for medical record review, 21 reported severe, potentially preventable adverse events in addition to the 11 such events recorded in the medical record. Caution The surveys took place 6 to 12 months after discharge. Implication Postdischarge patient surveys and medical record review are complementary ways to detect adverse events in the hospital. The Editors Improvements in patient safety depend in part on the ability to measure and monitor injuries to patients. The Institute of Medicine has called for building a national health information infrastructure that would include data about adverse events (1), but we lack consensus about the best way to measure adverse events. Common options for reporting adverse events include voluntary and mandatory reporting by providers, automated surveillance of clinical data, administrative data, and record review. Each of these methods has weaknesses (14). The medical record is often preferred because of its rich clinical content, but medical record review is expensive, documentation can vary among different clinical settings, and underreporting may occur because of concern about medicolegal liability (5). In recent years, patients have reported about the quality of care in various settings (69). Patients may also be an important source of information about hospital safety. Many hospitals survey patients after discharge to assess their satisfaction with care, so it may be relatively easy to add questions about safety events or symptoms that may suggest the occurrence of an event. Before embarking on such an effort, it would be useful to know what would be gained from the extra effort required. In other words, can patients identify safety events that are not described in the medical record? If so, what are the characteristics of those events? We assessed the concordance of adverse events detected by patient interviews with those detected by medical record review, and then characterized the adverse events that were detected by interview but not by medical record review. Methods Sample The sample included patients age 18 years or older who were hospitalized for medical or surgical treatment and were discharged from Massachusetts hospitals from 1 April to 1 October 2003. We devised a 2-stage probability sample. In the first stage, we grouped the 71 acute care hospitals into 4 strata on the basis of size. We selected the 5 largest hospitals with certainty and allocated the remaining 66 to 3 groups of 22 on the basis of size. Five hospitals were selected from each of the 3 remaining strata, for a total of 20 hospitals. Before data collection began, 4 of the original 20 hospitals withdrew because of resource constraints, leaving 16 hospitals. In the second stage, we randomly selected 6003 patients from the hospital discharge lists to achieve a final interview target of approximately 3000 patients after projected exclusions and nonresponse. Data Collection Protocol Hospitals supplied patients' names, contact information, age, dates of hospitalization, and discharge status. Eligible patients received a mailing with a toll-free number to opt out of the interview. Staff at the Center for Survey Research at the University of Massachusetts, Boston, made at least 6 calls before classifying someone as a nonrespondent. Because of delays in obtaining patient lists from the hospitals, interviews took place 6 to 12 months after discharge. Patients gave informed consent orally but had to provide a signed release form to allow us to review their medical records. The study was approved by the institutional review boards at Partners HealthCare System, the Massachusetts Department of Public Health, and each of the study hospitals. We created a firewall to prevent state staff from accessing identifiable data. Questionnaire We developed a draft questionnaire after holding focus groups. Initial items were cognitively tested with 10 respondents, followed by a field pretest with 32 persons. In its final form, the interview took about 20 minutes to complete. The first series of questions assessed 4 common categories of hospital treatment or management, including hospital staff administering medicines brought from home, new medicines given in the hospital, diagnostic tests, and surgery. Patients were asked to describe any negative effects or complications while hospitalized. To ensure completeness, we asked a second series of questions about 11 specified complications and injuries, including heart attack, stroke, uncontrolled bleeding, rash, and others. We recorded open-ended responses to both sets of questions. As a supplemental inquiry, we sought information about events discovered after the patient left the hospital but were related to care provided during their stay. Complications, such as wound infections, which often manifest after discharge, are potentially of interest to quality improvement efforts. Thus, we asked whether a complication they reported had occurred before, during, or after the hospital stay and whether they had to go back to any hospital or emergency room specifically because of complications associated with that hospital stay. We also searched the free-text descriptions. We studied these postdischarge adverse events separately from the other interview events and did not use them in our analyses of concordance between interview and medical record review. Medical Record Review We identified adverse events by using a structured implicit review tool adapted from the methodology used by the Harvard Medical Practice Study and others (1013). We defined adverse events as unintended harm to the patient by an act of commission or omission rather than by the underlying disease or condition of the patient (1). We created a prespecified list of 18 types of events (for example, hospital-acquired infections, adverse drug events) on the basis of a previous study (13) and created explicit evidence-based clinical case definitions to maximize reliability of judgments. The full list of events and the criteria set are available from the authors. Clinically experienced registered nurses received training by the investigators. The nurses identified adverse events from medical records by using the clinical case definitions and prepared case summaries for physician reviewers to evaluate. Nurses also recorded patients' major diagnostic code; length of stay; and comorbid conditions, which were summarized by a Charlson score (14). Physician Review Physician reviewers classified events and scored severity and preventability by using parallel methods for both the medical record reviews and the patient interviews. We analyzed only events confirmed by physician review. To classify interview events, we flagged patients' affirmative responses to queries about possible complications. Two physicians coded the first 31 cases together to clarify definitions. Next, both reviewers coded 120 reports to assess interrater reliability. Differences were resolved by consensus. One reviewer coded the remaining reports. Reviewers rated their confidence in the validity of the patient report (little or none, modest, strong, or virtually certain). Reviewers chose little or none if the patient report had too little information to adequately understand or interpret it, if it was internally inconsistent, or if the description was not clinically plausible. To classify medical record events, 2 physician reviewers (not authors) received training similar to that of the nurses. They independently assessed the case summaries and conferred to attempt consensus. By using this process, they always reached consensus without needing a third physician review. Reviewers classified events according to severity (life-threatening, serious, clinically significant, or trivial or insignificant) and preventability (definitely, probably, probably not, or definitely not) by using written guidelines. An example of a life-threatening event is a nosocomial urinary tract infection that leads to septic shock. Serious incidents generally involve an intervention and prolonged hospitalization (for example, a wound infection that requires debridement). Clinically significant events are more transient (for example, iatrogenic pneumothorax with minimal hemodynamic changes or an adverse drug event causing transient laboratory abnormalities). Trivial or insignificant events include minor injuries, such as pain at a venipuncture site or postural dizziness. Reviewers judged the preventability of the event on the basis of an assessment of the likelihood that it resulted from a clinical intervention that was inconsistent with current clinical practice or was a complication that could have been anticipated, rather than an anticipated manifestation of an underlying disease process. We used the statistic to characterize interrater reliability on the physician scores for the interview-based events (before they conferred about different scores), and weighted for ordinal variables. The scores were excellent for the presence of adverse events (0.85) and adverse drug events (0.97), type of incident (0.91), and category of hospital care (0.84). Agreement was good for preventability (0.71), fair for severity (0.45), and poor for confidence in the report (0.35). For record review, the intrarater and interrater reliability (between nurses and physicians) was reasonably good (0.61 to 0.96, depending on the measure) and was similar to previous adverse event studies that used physicians to review records (13, 15, 16). Medical Record Repeated Review To assess the frequency of missed or unrecorded medical record