Improvement Interventions Are Social Treatments, Not Pills

Even the most effective medical treatment (test, drug, or procedure) provides its full benefit only if it is applied to the right patient, in the right way and at the right time, every time. Recognition of the unfortunate reality that effective treatments are often applied inconsistently and inappropriately (1) has spurred development of a science of health care improvement. Derived in part from initiatives in industry, this nascent discipline is essentially a systematized version of experiential learning (2), informed by a philosophy that encompasses an understanding of human organizations, the nature of variation, and aspects of human psychology (3). Improvement interventions, like pills, can change clinical outcomes; unlike pills, however, they do so by applying innovative social treatments that change the way health care is organized and delivered, thereby narrowing the clinical knowledgeperformance gap. High-level evidence about efficacy and safety is lacking for many (probably most) improvement innovations being adopted into practice, which violates the basic canon of evidence-based medicine. In this issue, Goldman and colleagues (4) provide a piece of that missing evidence in their study that evaluated an improvement intervention using study methods at least as rigorous as those used in establishing the efficacy of new pharmaceuticals. The intervention, which involved language-concordant predischarge education by registered nurses plus follow-up telephone calls, was designed to reduce the rate of return emergency department (ED) visits and readmissions after an index hospitalization in an older, low-income, ethnically and linguistically diverse group of patients. The study method is a thing of beauty; its beauty, unfortunately, is also its curse. First, the beauty. The study protocol met all of the criteria expected of today's clinical research gold standardthat is, a randomized, controlled trial with a carefully designed study protocol; meticulous randomization; concealed assignment to treatment groups; aggressive collection of outcome data; and sophisticated, P valuebased statistical analysis. The researchers adhered faithfully to every detail of that protocol. The results provide convincing quantitative evidence that the intervention did not affect the primary outcome (that is, it did not reduce the rate of return ED visits or readmissions). Now, the curse. The strict adherence to study protocols required for valid, P valuebased statistical inference (5) freezes interventions in their initial form, which interferes directly with improvement's fundamental mechanismthat is, application of sequential, experiential learning cycles, which are sometimes referred to as plandostudyact cycles. Human behavior is hard to change, and the initial version of an improvement intervention is, therefore, often ineffective. An effective (or best) version often emerges only after the first has been modified multiple times in response to repeated feedback about its effectiveness (or lack thereof). (This process distresses institutional review boards because they generally insist on reapproving research projects if the initial study protocol changes significantly.) Goldman and colleagues do not tell us about their initial program theory (6) (that is, the nuts-and-bolts explanation of why they expected their intervention to work). However, once they learned that their intervention was ineffective, they could have responded by changing the intervention in various helpful ways, if the requirement for strict adherence to the study protocol had not prevented them from doing so. First, for example, one third of study patients (4) had visited the ED in the 6 months before the index admission, suggesting the presence of a preexisting group of persistent high users of hospital services within the study population. The researchers could therefore have refocused the intervention (as they suggest) to concentrate it intensively on a high-risk subgroup of patients such as this. Second, patients in the treatment group received print educational materials before leaving the hospital. However, the high rate of health illiteracy in their study population (approximately 50%) very likely limited the effectiveness of those materials (which presumably consisted of conventional printed text, although we are told only that they were language-concordant). The researchers could therefore have begun using print materials in pictographic formats, which are known to be highly effective in increasing retention of, and adherence to, oral counseling (7). Third, judging from the fact that the word family appears only once in their entire report (in the Discussion section), the intervention apparently involved patients' families to only a limited degree. However, families play a crucial role in caring for patients at home (8), particularly patients who have been recently hospitalized. The researchers could therefore have modified the intervention (as they also suggest) to increase the participation of families in the transition of patients from hospital to home. The inherent incompatibility between the fixed-protocol study methodsseen as essential for knowing whether an intervention worksand iterative, experience-based changes to improvement interventionsseen as essential for making them workclearly presents improvement researchers with an uncomfortable dilemma; it is not cause for despair, however. At least 2 approaches can help to resolve it, although neither does so cleanly or completely. The first is to use mixed study methods (that is, supplement the quantitative methods of controlled trials with qualitative descriptions). Numbers are essential for demonstrating whether interventions work, but numbers are not explanations, which is why explanations provided by qualitative methods can be so useful in fleshing out the interpretation of trial results. The second approach is to use time-series methods, such as statistical process control, in assessing the effect of improvement interventions (9). Although less familiar to clinical researchers than traditional frequentist statistics, time-series methods not only free researchers from the constraints imposed by protocol adherence but also can detect meaningful changes in outcome quite quickly and with relatively limited study samples, and can predict future performance rather than simply estimate past performance. Importantly, time-series methods can also provide statistical evidence of efficacy that compares favorably with that from randomized, controlled trial methods (10). The elegant study by Goldman and colleagues (4) teaches us a little about the particular intervention they applied in their particular setting: It did not work. It teaches us a great deal, however, about the need to evaluate improvement interventions using study methods that are appropriate for the living, context- and time-dependent nature of social treatments, which is what they are, rather than methods appropriate for the inanimate, standardized nature of pills, which they decidedly are not.