How Do You Prove Quality Improvement?

Field: Health Professions (Health Sciences)

Along with most of medical care, the Journal of the American Geriatrics Society is responding to the growing enthusiasm for quality improvement and evidence-based practice. Although such concepts are inherently attractive, an interesting dilemma arises when the two intersect. The basic approach to continuous quality improvement involves a cycle that starts with identifying a problem, collecting data to assess its size and ramifications, intervening on a limited scale, reassessing to determine the effect of the efforts to improve, and then—if the small-scale intervention is judged successful—broadening the intervention to a wider arena.1 Clinicians eager to improve their quality want to learn from the experience of colleagues about what works, but the unresolved challenge is what level of evidence must be offered before one can claim success in improving quality and attributing the improvement to the intervention used. In practice, most quality improvement studies use fairly crude data; the improvers are focused on achieving a rough idea of whether their efforts have yielded change. They do not ordinarily require strong chains of causation. A quality improvement study might involve reviewing a sample of charts at one point in time to establish a baseline and then reviewing a new sample after the intervention to see whether performance or outcomes have improved. Those doing the study would likely not worry much about sampling issues or make a great effort to adjust for case mix differences between the two samples. If the second sample looks better than the first, they will be inclined to celebrate a victory and press on. A greater challenge is encountered when the improvers want to use the experience of a project as evidence that a given approach works. The improvers may aspire to produce evidence that the intervention used can be replicated and generalized to other settings. The rules of evidence now become more stringent; they are now the rules of research. They require careful attention to design flaws and concerns about sampling, measurement, and causal inference. The article in this issue by Caruso et al.2 illustrates the dilemma. Anticipating that they would like to publish their results, they took pains to document and analyze the changes over time in an effort to address concerns about research design, but some problems could not be overcome. The study's real-world setting is at once its strength and its weakness. The authors cannot control which patients come to the clinic or even how often. They rely on repeated cross-sectional measures of patients with the same diagnoses (some of whom are the same people and some not)—but not necessarily with the same levels of clinical or sociodemographic characteristics—to show a pattern of change in clinical performance over time. They accept selection and attrition that would confound a research study and try to argue that these effects are not likely to influence their findings. They assume that the overall effect of the intervention will trump possible case-mix differences. What level of proof should we require to say that improvement has been achieved? What level of proof that the intervention was the cause of improvement? What about generalizability? Should the proof required be less than that demanded of health services research in general? Most quality improvement projects are not planned to provide high-quality health services research data to support conclusions drawn. To be acceptable and feasible, the projects must be pragmatic and unobtrusive. The problems arise when we try to learn from these experiences. The quandary has been the focus of much debate recently.3-6 Those who wish to accelerate the pace of quality improvement argue that the researchers are fossilized in the amber of scientific method, which is obstructing further progress.7 Those who wave the flag of caution see themselves as defending hard-won principles of science and avoiding mistaken conclusions that might lead to widespread waste or patient harm.8 We suggest that the dilemma can be best addressed by clarifying two things: (1) What do we want to prove? (2) Who is the audience? Usually we wish to prove that improvement actually occurred, that the intervention was the cause, and that the intervention could be used to good effect in other settings (that it is generalizable). The audiences might be people internal to the institution (hospital or clinic) being studied (people familiar with the setting and the patient population) or an external audience with little or no knowledge of the institution. These considerations can be represented in a simple matrix, as shown in Figure 1. What needs to be proved to whom. Step 4 is a necessary precursor to Step 5. Both require rigorous definitions, clear accounts of what interventions were used, rigorous measurement, statistical testing, and careful consideration of the limitations of the study reported. (Step 6 may be asking more than most research studies can achieve.) Activities aimed at internal audiences (the activities in Boxes 1 and 2) do not require the same rigor. (The activity in Box 3 is ordinarily pursued only in large institutions to determine whether improvements achieved in one hospital service or clinic can be generalized to other parts of the organization.) However, this lack of rigor does not mean that the activities are less valuable than those in the second row of the table. Action and measurement in Boxes 1 and 2 are quality improvement for local purposes. The measurement relies on managerial data collection and analysis, similar to measurement for any other management purpose (e.g., measurement of financial performance, customer service, or rates of manufacturing production). The time series data are interpreted along with unstated, perhaps ineffable, knowledge of the setting and the people involved. The use of data, along with judgment informed by long experience, has been well described.9 If changing patient characteristics, attrition of patients, or other potentially confounding factors are not measured—and they ordinarily are not, the possibility of being fooled by the data remains; practitioners of quality improvement must remain alert to this possibility. Quality improvement, including the associated measurement, has been called “pragmatic science.”10 That term suggests that quality improvement is different from and inferior to real science. We agree that it is different but do not concede that it is inferior. Indeed, it need not aspire to be science at all. When well done, it is excellent management. The question then arises about when it is suitable to report the activity in Boxes 1 and 2 to an external audience. First, reporting Box 1 activity without a Box 2 conclusion is probably never valuable. Assuming, then, that one is reporting Box 1 and 2 activity, the short answer is that it is suitable when there is enough evidence for Box 1 to be represented as Box 4 and Box 2 as Box 5. Activity in Box 6 is extremely difficult to pursue successfully, and few randomized, controlled trials succeed well in proving generalizability. The distinctions between Boxes 1 and 4 and between Boxes 2 and 5 are not clearcut. In both cases, the two contrasted activities lie on a continuum. Well-meaning quality improvers and researchers may disagree about how much evidence is needed to prove that improvement was actually achieved and to prove that the intervention was actually the cause of the improvement. Florence Nightingale's masterful account of what was likely the first quality improvement study (during the Crimean War, 1853–1856) illustrates the point. The improvement achieved in hospital mortality during 1855 is beyond dispute, dropping from 42.7% of those treated to 2.2% in 5 months. However, she goes to some lengths to demonstrate that her interventions to improve sanitation and decrease crowding were the causes of the improvement rather than the improving “condition of the Army” (i.e., the improving health of the troops before they were wounded in battle), as was claimed by some Army officers.11 Whether she convinced the Secretary of State for War, to whom her report was addressed, we do not know. It is possible to do quality improvement while planning to report the results to an external audience. This approach gives us the best of both worlds (good management and good science), although the cost is often not justified in the eyes of the people responsible for the financial performance of the institution engaged in the work. In that event, let good management proceed ahead but accept that reporting to an external audience may provide no benefit to that audience and therefore that publication is not warranted, regardless of whether publication is contemplated in the management literature or the medical literature. The next step in this conversation is to enumerate and articulate the dimensions of proof that are needed to move from an internal audience with local, implicit knowledge to an external audience that lacks this knowledge. That discourse is underway.3-8, 12-15 Financial Disclosure: Neither Dr. Mosser nor Dr. Kane has any relevant connections with any industries and neither has research grant support for work on this topic. Author Contributions: Drs. Mosser and Kane collaborated in writing this editorial. Sponsor's Role: Not sponsored.