Value in Health Care: Key Information for Policymakers to Assess Efforts to Improve Quality While Reducing Costs
Published: 2011-07-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. has devoted an increasing proportion of its economy and federal budget to the provision of health care services, but high levels of spending do not guarantee good care. Policymakers, health practitioners, and others have implemented numerous health care interventions that make discrete changes in the organization of health care services in order to enhance the value of health care--that is, improve the quality of care while reducing costs. Examples include programs to reduce bloodstream infections and to coordinate patient care following hospital discharges. This report (1) examines the availability of evidence on the effect of selected interventions on quality of care and costs; (2) identifies key dimensions for assessing the strength of such evidence; and (3) examines factors that can facilitate the implementation and replication of health care interventions. GAO identified a broad and diverse set of health care interventions using published and unpublished sources. For 127 of those interventions, GAO analyzed responses to a questionnaire that it sent to persons knowledgeable about available information on the effect of that particular intervention on quality of care and costs. GAO's questionnaire also asked respondents to assess the relative importance of seven factors in the implementation and potential replication of the health care intervention. In addition, GAO consulted the methodological literature and experts on assessing evidence on the effects of health care interventions.
section_title: What GAO Found
paragraphs: About half of the respondents to our questionnaire reported some information on the effect of an intervention on both quality of care and costs--the two types of data needed to determine whether or to what extent a particular intervention enhanced the value of health care. Overall, the vast majority of our respondents reported at least some information on the observed effect of the intervention on quality of care. Relatively fewer--though still over half--of our respondents reported at least some information on the effect of the intervention on costs. Whether or not policymakers can rely on information that indicates an intervention enhances value depends on the strength of the underlying evidence about quality and cost effects. From studies on the effect of health care interventions on quality of care and costs, policymakers and others can assess the strength and limitations of available evidence along three dimensions. One, the credibility of evidence on the effect of health care interventions on quality of care and costs depends primarily on whether those studies apply rigorous study designs. Two, the applicability of the results of studies to a broader population depends on the extent to which the study population is representative of that larger population. Finally, the capacity of health care interventions for widespread replication can be examined in terms of the consistency of results obtained by each intervention across diverse health care organizational contexts. Respondents reported, generally by large margins, that leadership support as well as other factors, such as organizational culture and staff resources, significantly facilitated implementation. However, respondents were more divided when asked about the reported effect that health IT had on implementation, and most respondents reported that financial incentives were not a factor in the implementation of the intervention. A majority of respondents reported that each of these factors, with the exception of financial incentives, would be either very or somewhat important if one were to attempt to replicate the intervention as widely as possible. Progress in achieving greater value in the U.S. health care system will depend, in part, on the availability of information regarding the effect of interventions on quality of care and costs and on how policymakers and others assess and use that information. Information can guide the choices of policymakers among multiple interventions vying for support, but those decisions will have a sounder basis if the information meets certain criteria regarding its content and strength of evidence. At least some information on both cost and quality effects was available for about half of the interventions GAO examined. However, for many interventions the credibility of this information was put into question by widespread reliance on studies that did not incorporate rigorous designs that could isolate the effect of an intervention from other factors. We requested comments from the Department of Health and Human Services, but none were provided.

section_title: Letter
paragraphs: For many years the United States has devoted an increasing  proportion of its gross domestic product (GDP) and federal budget  to the provision of health care services. National health  expenditures rose from 7 percent of GDP—$308 billion—in 1970 to  16 percent—$2.2 trillion—in 2008. Over this same period, the  proportion of federal budget outlays devoted to health care  increased even more rapidly from 10.5 to 32.6 percent. Unless this  trend is reversed, spending on health care will consume an  escalating share of federal resources, leaving fewer and fewer  dollars for other national priorities.
Furthermore, high levels of spending do not guarantee good quality  of care. While resources are needed to provide quality care,  spending more to increase the number or technical complexity of  treatments provided does not always lead to a corresponding  increase in the quality of care. At the same time, studies have  documented that many U.S. patients receive care of inconsistent  quality as measured in terms of adherence to recognized standards  of practice and in terms of clinical outcomes.
Faced with these challenges, policymakers, health practitioners,  and others have looked for ways to enhance the value of our health  care by improving the quality of that care and at the same time  reducing costs. To promote greater value across a range of health  care settings, they have implemented numerous interventions that  make discrete changes in who delivers health care services, how  care is organized, or where care is delivered for a specified  population. Some of these interventions are designed to restructure  the process of health care in ways that guide the behavior of  clinicians in fairly defined ways; for example, hospitals have  implemented checklists in their intensive care units to help reduce  the incidence of bloodstream infections by ensuring more  consistent compliance with recommended procedures. Other  interventions focus on restructuring how providers are paid to  encourage them to produce greater value without trying to specify  what steps they should take to achieve that objective. For example,  payments have been structured so that providers earn more if their  patients experience high quality care while overall costs are held in  check. Still other interventions are designed to motivate or assist  patients to take actions that will enhance their own health and  thereby reduce their need for health care services, ranging from  self-management of chronic conditions such as diabetes or heart  failure to maintenance of recommended drug regimens. The 2010  Patient Protection and Affordable Care Act (PPACA) includes  multiple provisions that support one or more of these approaches to  enhance the value of health care.
The wide array of different interventions gives policymakers the  opportunity to help improve the value of U.S health care by  supporting those interventions for which there is good evidence that  they improve quality and reduce costs. To identify these  interventions, policymakers need information on the effects of  interventions on both quality of care and costs, and they need that  information to be credible. It is therefore important for policymakers  to be able to weigh the strengths and limitations of the evidence  that an intervention, on net, has led to positive changes in quality of  care and costs. Finally, interventions may vary in their potential for  replication; that is, their effects on quality of care and costs may  differ substantially among the organizations—such as hospitals and  physician practices—that attempt to implement them. Therefore,  assessments of the evidence for different health care interventions  will be more complete if policymakers consider the interventions’  effects on quality of care and costs across different contexts.  Policymakers may also find it useful to know what factors may  inhibit or facilitate the implementation and replication of  interventions across varied organizational contexts.
To assist policymakers in their efforts to enhance the value of  health care, you requested that we provide you with information  about health care interventions. This report (1) examines the  availability of information on the effect of selected health care  interventions on quality of care and costs; (2) identifies key  dimensions for assessing the strengths and limitations of available  evidence on the effect of interventions on quality of care and costs;  and (3) examines factors that can facilitate the implementation and  replication of health care interventions.
To examine the availability of information on the effect of selected  health care interventions on quality of care and costs, we drew on  multiple sources to identify a broad and diverse set of interventions  that related to value in health care. In addition to an extensive  literature review, we conducted a comparable examination of other,  nonbibliographic sources including a database on quality  improvement initiatives maintained by the Agency for Healthcare  Research and Quality (AHRQ) and materials from presentations at  research conferences. The latter two sources allowed us to include  interventions that had not yet been described in academic or  professional literature. From these various sources we selected for  review a set of 239 interventions that appeared to meet all of the  following criteria: (1) the intervention had implemented a discrete  change in the organizational structure or process of health care  delivery; (2) available descriptions of the intervention suggested  that it both improved quality and reduced health care costs (or held  one constant while improving the other); and (3) the intervention  addressed issues relevant to the U.S. health care system. (See  app. I for a more complete description of the sources and methods  we used to identify interventions.)
Because the documentation that we obtained on these  interventions varied widely in focus, substantive content, and date  of issue, we chose to collect detailed information from individuals  with expert knowledge of each intervention using a standardized  data collection instrument. Thus, we developed a Web-based  questionnaire that we sent to researchers who were primary  authors of articles identified in our literature search or prepared  other materials describing these interventions and their results. The  questionnaire included a mix of open-ended and closed-ended  questions that examined what information was available on the  effect of the intervention on quality of care and costs as well as  what factors had facilitated or impeded implementation and  replication of the intervention. With respect to quality of care, we  asked respondents to describe up to five key measures that were  used to assess the effect of their intervention on quality of care and  the magnitude of change observed in those measures relative to a  baseline or a control group. Regarding costs, we asked  respondents to report the amount of any savings attributable to the  intervention as well as the methods and information used to  calculate those savings.
To identify key dimensions for assessing the strengths and  limitations of available evidence on the effect of interventions on  quality of care and costs, we reviewed the relevant methodological  literature on conducting systematic reviews and evaluations of  health care interventions. We also consulted with several subject  matter experts and obtained their reaction to the set of criteria that  we identified through that review to help policymakers critically  assess the information presented to them on value-enhancing  interventions. It was beyond the scope of this engagement to apply  this set of criteria to individual interventions. However, drawing on  the documentation collected in identifying interventions related to  value, we were able to categorize the types of study designs  employed by studies that examined the interventions for which we  received responses to our questionnaire.
With respect to factors that facilitated or impeded implementation  and replication of the intervention, we identified from relevant  literature seven factors that have been found to facilitate or impede  efforts to change the organizational structure and process of health  care delivery: leadership support, organizational culture, staff  resources, health information technology (IT), availability of tools  and activities to standardize care, financial resources, and financial  incentives. We asked our questionnaire respondents to assess the  extent to which each of these seven factors facilitated or impeded  the implementation of their intervention and the expected degree of  importance that each of these factors could have on attempts to  replicate the intervention to the widest scale possible. To gain  contextual understanding of how these factors affect  implementation and replication, we asked respondents to provide a  narrative description of how each factor facilitated or impeded  implementation of the intervention and why these factors would be  important for widespread replication, respectively. (See app. I for a  more extensive description of the information collected through our  questionnaire.)
We received usable responses for 127 of the 239 selected  interventions. These 127 interventions applied a broad range of  value-enhancing strategies in different health care settings and  among diverse patient populations, including enhanced  management of patients with chronic conditions and coordination of  care across multiple providers (see app. II for a complete list of the  different types of interventions included). Although our efforts to  identify relevant interventions were extensive, we could not ensure  that every intervention meeting our selection criteria had been  identified. As a result, our findings are limited in scope to the  interventions for which we received completed questionnaires and  cannot be generalized to all value-enhancing health care  interventions.
We conducted this performance audit from April 2009 to July 2011,  in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Background
paragraphs: The methodological literature provides insight into conducting  systematic assessments of evidence for health care interventions  that change the delivery or structure of care. Furthermore, the  literature on organizational change is pertinent to understanding the  key factors that can facilitate or impede implementation and  replication of such health care interventions.
Applied social science research has developed a core set of  methodological questions and approaches for assessing the effect  of programs or other interventions on a wide range of organized  activities. They address two key issues: how best to determine the  independent effect of a program or intervention and how best to  generalize from the results obtained from one or more studies to  broader populations of interest. A number of organizations have  developed more specialized guidance for applying these general  methodological principles to health care interventions. For example,  the Effective Practice and Organisation of Care Group (EPOC) is a  component of the Cochrane Collaboration—an international  network of individuals who analyze the effect of health care  interventions—which focuses on interventions that change the  practice of care and the delivery of health care services. EPOC  provides guidance to researchers on how best to prepare  systematic reviews of such interventions in order to synthesize the  information available in multiple studies. AHRQ’s Effective Health  Care Program (EHC) and the Grading of Recommendations  Assessment, Development and Evaluation (GRADE) working group  have developed similar guidance, though both of these efforts focus  more on assessing alternative medical treatments rather than  alternative approaches for organizing health care services.
Research on organizational change has identified certain factors as  key contributors to successful implementation of health care  interventions for quality improvement. For example, the literature  consistently cites leadership support as essential for successful  implementation of quality-of-care interventions within health care  settings. Further, it describes the role of leaders in promoting the  adoption of interventions by their organizations, in winning  acceptance among affected staff members for the changes those  interventions entail, and in marshalling sufficient resources for the  intervention to succeed. In addition, this literature has shown that  organizations vary in their attitudes, beliefs, and values, and that  this “organizational culture” can either promote or inhibit change.  Organizations tend to achieve quality of care improvement more  readily if they have a culture with such characteristics as  receptiveness to change, placing high value on ensuring the quality  of care provided, and prizing innovation with a willingness to take  risks. The literature also cites the role of infrastructure factors, such  as the sufficiency and appropriateness of staff resources and the  adequacy of existing health information technology (health IT)  systems, in the successful implementation of quality improvement  interventions. Another factor cited in the literature is the availability  of previously developed tools and procedures for standardizing  health care processes—such as checklists or guidelines—as well  as other types of technical assistance that can facilitate the  implementation of a given intervention. Additionally, the literature  has pointed to financial factors that affect the implementation of  interventions for quality improvement, including both the level of  financial resources needed to sustain an intervention and the use of  financial incentives to promote quality enhancement activities.  Financial incentives represent a particular application of financial  resources that involve the contractual or other provisions that  determine how much health care providers are paid and for what.  Such financial incentives affect who benefits from and who pays for  the cost of an intervention. This in turn can facilitate or impede the  implementation and replication of interventions.
section_title: Respondents Reported Basic Information Needed to Assess Value Available for About Half of Selected Interventions
paragraphs: About half of the respondents to our questionnaire reported basic  information on the effect of their intervention on both quality of care  and costs—the two types of data needed to determine whether or  to what extent a particular intervention enhanced the value of  health care. Overall, the vast majority of our respondents reported  at least some information on the observed effect of their  intervention on quality of care. Relatively fewer—though still over  half—of our respondents reported at least some information on the  effect of their intervention on costs.
The ability of policymakers to identify interventions that  substantially improve quality and reduce costs depends on the  availability of basic information on the size of the effect of an  intervention on both quality of care and costs. These are the two  types of data needed to determine whether or to what extent a  particular intervention enhanced the value of health care. Just over  half of the respondents to our questionnaire reported such basic  information on their interventions. Sixty-four of 127 respondents  reported information on both improvements observed in at least  one quality measure and a specific amount of cost savings (see  table 1). For the remaining interventions, the missing information  most often concerned the effect of the intervention on costs.  Furthermore, even fewer respondents, 45, reported improvements  observed in at least one quality measure and a specific amount of  cost savings that accounted for the costs of implementing their  intervention—net cost savings.
Compared to information on both quality of care and costs,  information on the effect of selected interventions on quality of care  alone was more frequently reported. The vast majority of  respondents to our questionnaire reported at least some  information on the observed effect of their intervention on quality of  care. Specifically, 114 of 127 respondents reported improvements  in one or more measures used to assess the effect of their  intervention on various aspects of care quality. Of these,   112 respondents reported a specific magnitude of improvement  observed in at least one quality measure in terms of a percentage  change or other quantitative measurement. Additionally,   2 respondents reported improvement in at least one quality  measure, but did not report a specific magnitude of improvement. In  contrast, the remaining 13 respondents did not report sufficient  information to determine whether their intervention had any effect  on quality of care. Six of 127 respondents described one or more  measures used to assess the effect of their intervention on different  aspects of care quality, but did not report a magnitude of  improvement observed in these measures. Seven respondents did  not report any information on the measures used to assess the  effect of their intervention on aspects of care quality.
Respondents reported that the effect of their intervention on quality  of care was assessed using a range of measures that generally fell  into five broad types reflecting different aspects of care quality (see  table 2). Respondents most frequently described one or more  quality measures that were used to assess the effect of their  intervention on outcomes resulting from care. Specifically,   82 respondents reported that the effect of their intervention on  quality of care was assessed using outcome measures such as  patient mortality, the overall physical and emotional health of a  patient, or the level of stress reported by a patient caregiver. In  addition, 56 respondents described one or more measures that  assessed the effect of their intervention on the amount of health  care services consumed. These measures included the length of  hospital stay, the number of emergency department visits, and the  number of hospital readmissions for a specified population. Forty- four respondents described measures that assessed the effect of  their intervention on processes of care. Process-of-care measures  assess the extent to which the care provided to a patient was  appropriate based on current professional knowledge and the  particular circumstances. For example, process-of-care measures  could examine whether diabetes patients had received foot exams,  eye exams, and regular glucose monitoring at specified intervals.
Fewer respondents described measures that assessed quality in  terms of the experience of a patient or caregiver or the structure of  care.
Although the information provided by any one type of quality  measure is limited, most of our respondents reported that the effect  of their intervention on quality of care was assessed using more  than one type of quality measure. Each type of quality measure  offers insight into a particular domain of quality such as outcomes  of care, processes of care, or experience of care. Just 41  respondents reported that only one type of measure was used to  assess the effect of their intervention on quality of care (see fig. 1).  For most—79—of the interventions in our review, respondents  reported that the effect of their intervention on quality of care was  assessed using measures belonging to two or more different types  of quality measures, thereby providing a broader perspective on the  effect of the intervention on quality of care.
Somewhat fewer respondents to our questionnaire reported  information on the effect of their interventions on costs than quality  of care. Specifically, 72 of 127 respondents reported a specific  amount of change in costs—cost savings. Respondents most  frequently reported that costs were assessed by calculating the  total dollars saved or the average dollars saved per person  annually. Respondents less frequently reported that costs were  assessed by calculating the financial return on investment,  percentage change in total health care costs per patient, or an  alternative cost metric such as dollars saved per member per  month for patients participating in a certain health care plan. In  contrast, the remaining 55 respondents did not report sufficient  information to determine whether their intervention had any effect  on costs. Nine of 127 respondents reported that costs were  assessed, but did not report a specific amount of cost savings.  Forty-five respondents reported that cost savings were not  assessed, and one respondent did not report any information on  whether cost savings were assessed.
Most, but not all, of the respondents who reported a specific  amount of cost savings stated that these cost savings accounted  for the costs associated with implementing the intervention. Among  the 72 respondents who reported a specific amount of cost savings,  51 respondents reported net cost savings that took account of  implementation costs; another 20 respondents reported gross cost  savings that did not take implementation costs into account. When  asked to provide additional detail on their implementation cost  calculations, 35 respondents reported that the cost savings took  account of both start-up costs associated with developing and  initially implementing the intervention as well as ongoing costs  associated with operating and maintaining the intervention over  time. Two respondents reported that cost savings took account of  start-up costs but not ongoing costs to maintain the intervention,  and 19 reported that cost savings took account of ongoing costs but  not start-up costs.
The interventions we reviewed also varied in the extent to which the  reported cost savings attributed to them were based on information  directly related to the intervention. Forty-nine respondents reported  that cost savings were calculated using only data that were  collected specifically to assess the effect of their intervention on  costs. In contrast, 26 respondents reported that cost savings were  calculated using a mix of data that were collected specifically to  assess the intervention and data from a secondary source such as  published literature or a national database. For example, one  respondent reported cost savings attributable to an intervention  designed to improve patient self-management of asthma based on  data that were collected on changes over time in the actual number  of health care encounters for patients enrolled in the program and  the estimated costs for those encounters derived from national  averages for several types of health care services such as hospital  days or emergency department visits. While data from secondary  data sources may provide otherwise missing information needed to  estimate the cost savings achieved by an intervention, the  relevance of such secondary data to that particular intervention  may be open to question, which makes the accuracy of the cost  savings estimate more uncertain.
section_title: The Strength of Evidence on the Effect of Interventions Can Be Assessed along Three Dimensions
paragraphs: Policymakers and others can assess the strength and limitations of  available evidence from studies on the effect of health care  interventions on quality of care and costs along three dimensions.  One, the credibility of evidence on the effect of health care  interventions on quality of care and costs depends primarily on  whether those studies apply rigorous study designs. Two, the  applicability of the results of studies to a broader population  depends on the extent to which the study population is  representative of that larger population. Finally, the capacity of  health care interventions for widespread replication can be  examined in terms of the consistency of results obtained by each  intervention across diverse organizations. Appendix III provides a  more detailed explanation of what makes some study-design types  more rigorous than others and appendix IV presents a list of key  questions that describe the information that policymakers can look  for to assess the evidence provided by particular studies along  these three dimensions.
For policymakers and others, the benefit obtained from basic  information on the effect of interventions on quality of care and  costs depends in large part on the strength of that evidence.  Information based on weak evidence can provide policymakers a  misleading indication of an intervention’s potential to enhance  value. For example, the direction and magnitude of the changes in  quality of care and cost reported for the 127 interventions examined  through our questionnaire could deviate substantially from the  actual impact of those interventions, depending on the  characteristics of the studies that generated that reported  information. To determine what information has the kind of  evidentiary support that they can rely on, policymakers can assess  the strengths and limitations of studies that examine health care  interventions of interest along three broad dimensions.
The first of these dimensions is the credibility of evidence that  attributes any changes in quality of care and costs to those  interventions. The methodological experts we consulted uniformly  emphasized the primacy of study design in determining the  credibility of evidence on the effect of health care interventions on  quality of care and costs. Observed changes in quality of care and  costs that one might attribute to a health care intervention may in  fact be due in large part to the effect of a wide variety of other  factors. The choice of study design type is critical because rigorous  designs have the capacity to isolate the effects of a health care  intervention from other factors that may affect changes in quality of  care and costs.
The methodological literature we reviewed identifies several  different study design types that have sufficient rigor to isolate the  effect of interventions on quality of care and costs. They include  randomized controlled trials (RCTs), interrupted time series studies,  and controlled before and after studies. RCTs and controlled  before and after studies both use control groups—consisting of  study participants who are not exposed to the intervention—to  adjust for the effect of other factors besides the intervention.  Interrupted times series studies do not use control groups; instead  they rely on analyzing data collected at multiple time points both  before and after an intervention is implemented to adjust for other  factors. (See app. III for more information on how these study  design types isolate the effect of an intervention.)
In contrast, according to the methodological literature we reviewed,  some other types of study designs lack the capacity to isolate the  effect of a health care intervention from that of other factors. For  example, a simple pre/post study that assesses quality of care and  costs once before an intervention is implemented and a second  time after implementation of the intervention has no mechanism  analogous to a control group to take account of the effect of other  factors. The same is true for post-only studies that rely entirely on  data collected after an intervention was implemented. With studies  using these types of designs, there is no way to determine how  much of the difference observed between the pre and post  measurements, or among any groups following an intervention, was  due to the intervention and not to other factors. Consequently, such  studies will not provide policymakers credible information about the  extent to which the intervention itself affected both quality of care  and costs.
Table 3 describes key distinguishing characteristics to help  policymakers identify the type of study design employed in a study  of an intervention.
Among studies addressing the effect of health care interventions on  quality of care and costs, a range of rigorous to weak design types  are used. For example, among the 127 interventions for which we  received responses to our questionnaire, we found 22 interventions  with studies involving RCTs and another 11 interventions assessed  using controlled before and after studies. However, for a  substantially larger number of the 127 interventions, the studies we  identified employed the types of study designs that do not isolate  the effect of the intervention from other factors. Specifically, the  results for 67 interventions were based on pre/post studies, and  another 19 were based on post-only studies of one kind or another.  In this one, diverse set of interventions that we reviewed,  policymakers could find credible evidence based on rigorous study  designs concerning the effects of certain interventions on quality of  care and costs; however, for many other interventions such studies  were lacking.
In addition to study design, the methodological literature we  reviewed emphasized the importance of how a study is conducted.  Even rigorous study designs can lose their capacity to isolate the  effect of an intervention on quality of care and costs if researchers  do not adhere to the requirements of those designs. Thus,  assessments of the strengths of study results should consider how  well the study design was implemented.
One component of a study’s implementation that policymakers can  examine involves the selection and management of control groups  used in the study. In order to isolate the effects of an intervention,  the control group has to be equivalent to the treatment group— except for the latter’s exposure to the intervention. According to the  methodological literature we reviewed, that equivalence can be  compromised in a number of ways. In the case of RCTs, for  example, allocation to treatment and control groups may not be  truly random if there are flaws in the process for assigning study  subjects to those groups. Moreover, for both RCTs and controlled  before and after studies, losing a disproportionate number of study  participants from either treatment or control groups can also  undermine their equivalence.
Another component of a study’s implementation that policymakers  can examine concerns the measures and procedures adopted for  data collection. According to the methodological literature we  reviewed, a study will produce stronger evidence when it employs  measures that are recognized as valid and reliable. For example,  central line-associated bloodstream infections can be tracked using  a surveillance measure developed by the Centers for Disease  Control (CDC) or with less labor-intensive measures that draw on  administrative data. Clinicians consider the CDC measure to be the  most valid and reliable measure for this type of infection because it  calls for laboratory confirmation of identified infections and it  accounts for varying risks of infection based on the number of days  that a central line catheter is in place. In addition, the data for  those measures should be collected at the same time and in the  same way from all groups in the study. Any systematic  inconsistencies in how data are collected for a study can skew the  results.
If a study produces credible evidence that a health care intervention  has a positive effect on both quality of care and costs within the  population it examined, a second dimension that policymakers and  others can assess concerns the scope of that effect—for what  broader populations or groups are the results applicable?  Applicability depends on the representativeness of the study  population for a broader population of interest. The methodological  literature identifies two different approaches for establishing  representativeness: (1) randomly selecting the study population  from a known universe, or (2) examining the degree to which a  study population matches a given broader population on  characteristics relevant to the intervention. The first approach,  random selection, intrinsically makes the study population  representative of the particular universe from which it was selected  and the study results applicable to that population.
The second approach for establishing representativeness— examining the extent of similarity between the study population and  a broader population of interest—can be used by policymakers  whenever the study population was not chosen randomly or the  broader population of interest to policymakers is not the universe  from which the study population was selected. Policymakers can  assess the degree of similarity between the study population and a  broader population through an examination that focuses on two  issues: (1) identifying characteristics where the study population  and broader population of interest differ and (2) assessing whether  any differences found could influence the effect of the intervention  on quality of care and costs (see app. IV ).
Major differences between a nonrandomly selected study  population and a broader population of interest to policymakers  should raise questions about the applicability of the study’s results  for that broader population. For example, an intervention to improve  care coordination for patients with diabetes might be implemented  and assessed in a few academic medical centers. In that situation,  the representativeness of the study population for all patients with  diabetes could come into question on at least two counts—the kind  of care provided in an academic medical center might well differ  from that usually provided by community-based providers and the  patients treated by academic medical centers might have a higher  level of severity than diabetics treated elsewhere. If patients in the  study received a different overall set of services, that could affect  the impact of the intervention on those patients even if the  intervention itself were implemented the same way for the two  populations. Similarly, an intervention could have a more  pronounced effect on patients with a higher level of severity, or the  intervention might work less well for such patients. Thus, to  establish the applicability of the study results to a broader  population of diabetic patients, studies of the intervention would  need to provide evidence that the differences between the study  population and the broader population of diabetics would not affect  the performance of the intervention.
A third dimension on which policymakers and others can assess  the strength of evidence for health care interventions concerns the  capacity of an intervention for replication across diverse  organizations. Because organizations vary across the factors that  affect the implementation of health care interventions, including  leadership, organizational culture, and staff and financial resources,  a particular intervention may work more or less well depending on  the organizational environment in which it operates. As a result,  some organizations may be more receptive to a particular value- enhancing intervention than others. That, in turn, can make it more  difficult to take an intervention that proved successful in a small  number of organizations and replicate it widely across many others.  However, some interventions have produced positive results on  quality of care and costs in a range of different organizations, which  suggests that they may be less sensitive to varying circumstances  across organizations.
According to the methodological literature and experts that we  consulted, certain information can provide the basis for an  assessment of the consistency in an intervention’s effects on quality  of care and costs in different organizations. Specifically, this  information concerns the number of different organizations where  the intervention has been implemented, the degree of diversity  exhibited by those organizations, and the consistency in observed  changes in quality of care and costs across those organizations.  However, such information would not be available for assessing the  consistency of results across diverse organizations if an  intervention has been implemented in only a few different  organizations, or in multiple organizations that are generally quite  similar. That is also the case if studies only analyze and report  changes in quality of care and costs attributed to an intervention in  the aggregate, rather than separately for the different organizations  that implemented it.
On the other hand, for interventions that have been implemented in  multiple, diverse organizations, and their results analyzed  separately at the different organizations, it is possible for  policymakers to compare the results of the intervention across  those organizations to examine the consistency of the intervention’s  effect. To the extent that those interventions consistently produce  positive effects on quality of care and costs among diverse  organizations, that provides evidence of their capacity for  widespread replication. For other interventions, if data on the  changes in quality of care and costs across the different  organizations indicate a lack of consistency in outcomes, that  provides evidence of a more restricted capacity for replication.
section_title: Leadership Support and Other Factors Reported As Important for Both Implementation and Replication of Interventions
paragraphs: Respondents to our questionnaire reported, generally by large  margins, that leadership support as well as other factors, such as  organizational culture and staff resources, significantly facilitated  implementation. However, respondents were more divided when  asked about the reported effect that health IT had on  implementation, and most respondents reported that financial  incentives were not a factor in the implementation of their  intervention. A majority of respondents reported that each of these  factors, with the exception of financial incentives, was expected to  be either very or somewhat important if one were to attempt to  replicate their intervention as widely as possible.
Taking account of factors that prior research has shown tend to  facilitate or impede the implementation and replication of  interventions may enhance efforts by policymakers and others to  promote the adoption of interventions across varied organizational  contexts. In examining the relative impact of seven factors identified  in our literature review, we found that respondents to our  questionnaire reported, generally by large margins, that five of the  seven factors significantly facilitated implementation of their  intervention. Health IT and financial incentives were the exceptions.  Leadership support was the factor that the largest number of  respondents reported as having significantly facilitated  implementation of their intervention (see table 4). When asked to  describe how leadership support facilitated implementation,  respondents frequently explained that a leader who visibly  prioritized and endorsed the intervention, allocated necessary  resources, and championed the development and implementation  of the intervention and drove necessary organizational or  behavioral changes facilitated the implementation of the  intervention. Respondents also explained that having champions,  specifically clinicians, was a key factor in encouraging cooperation  and participation in the intervention by staff, especially fellow  clinicians. The prominent role attributed to leadership in  implementing the many different types of interventions in our  sample suggests that policymakers will have greater success in  implementing and replicating interventions to the extent that they  can take steps to ensure that strong leadership is in place before  interventions are initiated.
Respondents typically reported that a combination of additional  factors along with leadership support significantly facilitated  implementation of their intervention. The 92 respondents who  reported leadership support as having significantly facilitated  implementation, reported, on average, another three factors as  having significantly facilitated implementation. Of the 86  respondents who reported at least one factor in addition to  leadership support as significantly facilitating implementation, more  than half reported staff resources (60), organizational culture (55),  and the availability of other tools (50), respectively, as having  significantly facilitated implementation. Nearly half (42) reported  that financial resources, in addition to leadership, significantly  facilitated implementation. Just six respondents reported leadership  support and no other factor as having significantly facilitated  implementation.
In contrast to the five factors that a clear majority of respondents  reported having facilitated implementation of their intervention,  respondents were more divided on how health IT affected  implementation, as shown in table 4. Health IT had the highest  number of respondents, compared to the other factors, that  reported the factor impeded implementation of their intervention.  Further, a substantial group of respondents reported that health IT  was not a factor. On the other hand, nearly half of respondents  reported that health IT either significantly or somewhat facilitated  implementation of their intervention. Respondents frequently  explained that health IT facilitated implementation of their  intervention by enhancing the exchange of information and  communication across providers or organizations, facilitating the  collection of data or the evaluation of the intervention and improving  the efficiency and productivity of staff. Of those who reported that  health IT impeded implementation, respondents commonly cited  the limited functional capacity of existing systems or the lack of  interoperability across settings as impediments to successful  implementation. Other respondents explained that the general  lack of health IT altogether acted as a barrier that impeded  implementation. Variation in the role of health IT across different  types of interventions does not appear to explain the mixed  assessment of this factor; as respondents for each of the  intervention types included in our sample—with two exceptions— were similarly divided on how health IT affected implementation.  However, proportionately more respondents for care coordination  or transitions of care interventions as well as care-process- improvement interventions reported health IT as having facilitated  implementation compared to respondents for other types of  interventions. This result suggests that as policymakers consider  different health care interventions, implementation of some of their  options will depend more heavily than others on having  appropriately configured health IT in place.
Financial incentives were most often reported as not a factor.  Slightly more than half of our 127 respondents reported financial  incentives—as distinct from the related, but broader, financial  resources factor—as having not been a factor in implementation of  their intervention. The exception was for the two types of  interventions for which financial incentives were an integral  component—provider payment restructuring and insurance  redesign—where respondents most often reported financial  incentives as having significantly facilitated implementation. When  asked to explain how financial incentives facilitated or impeded  implementation most respondents simply provided a description of  the incentives they used to implement the intervention, such as  payments to providers or patients to participate in the intervention.  However, a few respondents explained that the expected cost  savings generated from the intervention was an indirect incentive to  implement the intervention while other respondents explained that  incentives within existing payment systems, or the lack thereof,  affected implementation. While the implementation of many  interventions included in our sample may not have been affected by  financial incentives, current means of paying for health care, such  as fee-for-service payment structures, may have hindered the  successful implementation of other interventions.
Much as they had reported regarding the implementation of their  intervention, nearly all respondents consistently expected that  leadership support would be very important if one were to attempt  to replicate their intervention as widely as possible (see table 5).  Leadership support was reported nearly unanimously by  respondents as being very important for widespread replication of  their intervention, paralleling respondents’ relatively consistent  assessment of the effect of leadership on implementation. In  addition, a clear majority of respondents expected that each of the  other factors—except for financial incentives—would be either very  or somewhat important for replication.
In contrast to the highly divided views health IT evoked from  respondents regarding its role in the implementation of their  interventions, it was reported by a substantial majority of  respondents as either very (48) or somewhat (48) important for  widespread replication. This could be an indication that, if health-IT- related impediments experienced when implementing the  intervention, such as the lack of interoperability across settings,  were ameliorated, health IT could be important to the successful  replication of some interventions. Similar to views expressed about  the implementation of care coordination or transitions of care  interventions, respondents for these types of interventions  commonly reported that health IT would be very important for  widespread replication more so than respondents for other types of  interventions.
Financial incentives was the factor that drew the most mixed  assessment from respondents with regards to its expected  importance for the widespread replication of interventions. Nearly  half of respondents indicated that financial incentives were not  important for widespread replication, which is similar to the view of  most respondents regarding the role of such incentives in the  implementation of their interventions. Another substantial group of  respondents (30) indicated that financial incentives would be very  important for replication. When respondents were asked to explain  why factors would be important for widespread replication,  respondents discussed financial factors more frequently than any  other factor. Respondents’ explanations about these financial  factors often concerned a misalignment of financial incentives  within existing payment systems that limited the attractiveness of  replicating interventions that seek to enhance value. For example,  some respondents noted that it would be difficult to replicate  interventions that involved providing additional services, such as  care coordination, under existing payment systems that typically do  not compensate providers for those services.
section_title: Concluding Observations
paragraphs: Our work suggests that progress in achieving greater value in  health care in the U.S. will depend, in part, on the availability of  information regarding the effect of different interventions on quality  of care and costs and on how policymakers and others assess and  use that information. Such information can guide the choices of  policymakers among multiple interventions vying for support, but  those decisions will have a sounder basis if the information meets  certain criteria regarding its content and strength of evidence. With  respect to content, information on the magnitude of an  intervention’s effect on both quality of care and costs is needed to  determine if an intervention has enhanced value. In the case of the  responses to our questionnaire on 127 diverse interventions, we  found that this basic level of information was reported as available  about half the time. With respect to the strength of evidence, the  most critical indication comes from the types of study designs used  to produce that information. There are a range of rigorous study  designs which can provide credible support for the attribution of  observed changes in quality of care and costs to a particular  intervention. Our review of studies associated with the 127  interventions examined by our questionnaire found that while a  number of studies employed rigorous study designs, a substantially  larger number employed weaker designs that could not isolate the  effect of an intervention from other factors. To the extent that  policymakers find and use information on health care interventions  that provides sufficient credible evidence on the effects of those  interventions on both quality of care and costs, they will be better  equipped to determine which interventions produce greater value in  health care. Our work also suggests that successful efforts to  encourage the widespread adoption of value-enhancing  interventions will need to take into account a complex mix of  factors, including leadership support, organizational culture, and  staff resources, that facilitate the implementation of health care  interventions across a wide range of organizational contexts.
section_title: Agency Comments
paragraphs: We requested comments from the Department of Health and  Human Services, but none were provided.
We are sending copies of this report to the Secretary of Health and  Human Services and other interested parties. In addition, the report  will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs  may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To examine the availability of information on the effect of selected  health care interventions on quality of care and costs as well as  factors that can facilitate the implementation and replication of  these interventions, we studied a diverse set of specific  interventions that seek to enhance the value of health care through  making changes in the way care is delivered. Specifically, these  interventions make changes in who delivers health care services,  how care is organized, or where care is delivered for a specified  population. To identify interventions for our study, we drew upon six  distinct sources to select a broad and diverse, though not  exhaustive, set of interventions that have been implemented in one  or more locations in the U.S. or abroad. These sources allowed us  to identify a wide range of value-enhancing strategies implemented  in different of health care settings, such as hospitals, integrated  delivery systems, and physician practices, over more than a   10-year time span, including interventions that have not been  described in academic or professional literature. We identified 828  interventions of potential relevance to our study through the  following six sources:    A review of relevant literature on health care interventions that  make changes in who delivers care services, how it is delivered,  or where it is delivered. We conducted several searches using  online databases, including Medline and ProQuest Health, to  identify articles on interventions that were published from 1999  to 2009.    A review of interventions contained in the Agency for Healthcare  Research and Quality’s (AHRQ) Health Care Innovations  Exchange (HCIE) as of August 20, 2009. The HCIE is a Web  site that acts as a repository for information on quality  improvement interventions and other innovative strategies to  improve health care submitted by their implementers. Many of  the interventions contained in the HCIE make changes in the  way health care is delivered and include information on cost as  well as quality.    A review of the relevant articles contained in the Tufts Medical  Center’s Cost-Effectiveness Analysis (CEA) Registry that were  published from 1999 to 2009. The CEA Registry is a  comprehensive database of health care cost-utility analyses that  examine the health benefits and costs of strategies to improve  health care. The CEA Registry contains articles from 45 peer- reviewed publications.
Interviews with experts on health care interventions associated  with organizations such as state governments, integrated  delivery systems, employer groups, and other countries.
Information on interventions that we identified from press  reports, select journal articles published after 2009, and  presentations at conferences.
Information on interventions submitted by their innovators or  evaluators to either the Senate Budget Committee or GAO.
To select interventions for inclusion in our study, we reviewed  source documents for each of the potentially relevant interventions  that we identified through our six sources. We selected 239  interventions that met the following seven criteria:    The intervention made a discrete change in who delivers health  care services, how care is organized, or where care is delivered.    The intervention targeted a population or problem that was  relevant to the U.S. health care system.    The intervention may have included health information  technology (health IT) as one of its components of change, but  health IT was not the intervention’s only component of change.    The primary goal of the intervention was not focused on  increasing access to care.    The intervention activities must fall within the health care  system.    The source document or documents for the intervention either  contained information or indicated that information is available  on the effect of the intervention on quality of care and its effect  on costs. Moreover, the source documents indicated that the  intervention enhanced the value of health care by meeting one  of the following three conditions: (1) increases quality of care  and reduces costs; (2) maintains quality of care and reduces  costs; or (3) increases quality of care and maintains costs.    The intervention was implemented in at least one health care  setting. Interventions that were studied by examining their  potential costs and benefits based on simulated outcomes  rather than analyzing data from their actual implementation  were excluded.
To collect information on the 239 health care interventions that we  selected for our study, we developed a Web-based questionnaire  that contained 22 open- and closed-ended questions on  interventions, their effect on quality of care and costs, and factors  that may affect their implementation and replication. We sent our  questionnaire to 235 individuals who participated in developing,  implementing, or evaluating each intervention. We identified these  individuals through the source documents that we used to select  interventions for our study. We received usable responses— responses that contained relevant information on the effect of the  intervention on quality of care, the effect of the intervention on  costs, or key factors that may affect implementation—for 127  interventions. We developed protocols for cleaning and analyzing  data that we received from questionnaire respondents. These  protocols included: identifying usable responses; reviewing source  documents to clarify responses; and, if necessary, contacting  respondents directly to obtain additional information on their  intervention.
To determine the availability of information on the effect of selected  health care interventions on quality of care, we analyzed data that  we collected from respondents through our questionnaire. We  asked respondents to describe up to five key measures used to  assess the effect of their intervention on quality of care and the  magnitude—a percentage change or other quantitative  assessment—of change observed in each measure described  relative to a control group that did not experience the intervention or  a baseline assessment made prior to implementing the intervention.  We conducted a content analysis on questionnaire responses to  determine the number of respondents who described one or more  key measures used to assess the effect of their intervention on  quality of care and the number of respondents who reported  improvements in those measures attributable to their intervention.
As part of our analysis on the availability of information on the effect  of selected health care interventions on quality of care, we  examined the types of quality measures respondents reported. We  conducted a content analysis on questionnaire responses to  determine what aspect of care quality—such as patient mortality,  hospital readmissions, or patient satisfaction with care—each  measure examined. We categorized each measure that  respondents described by type based on the aspect of care quality  it examined; for example, we categorized a measure that assessed  the effect of an intervention on patient mortality as an outcome  measure. We categorized quality measures into types that are  largely based on the measure domains laid out by AHRQ in its  National Quality Measure Clearing House. We did not include all  measure domains laid out by AHRQ in our analysis, because some  domains, such as access to care, fell outside of the scope of our  engagement. Moreover, measures that did not clearly specify which  aspect of care quality was assessed were categorized as  unspecified measures. We analyzed this information to determine  the types of quality measures used to assess the effect of each  intervention on quality of care.
To determine the availability of information on the effect of selected  health care interventions on costs, we analyzed data that we  collected from respondents through our questionnaire. We asked  respondents to report the type of cost savings, such as total dollars  saved or dollars saved per patient, calculated to assess the effect  of their intervention on costs and the specific amount saved for type  of cost savings calculated. We also asked respondents if their  reported savings accounted for costs associated with implementing  the intervention and what information was used to calculate those  savings. We determined the number of respondents who reported  calculating each type of savings and a specific amount saved for  those savings. Furthermore, we analyzed responses by finding the  number of respondents who reported accounting for costs  associated with implementing the intervention—net cost savings— and the type of information used to calculate those savings.  Additionally, we used this information along with information we  obtained through our analysis of quality measures to determine   (1) the number of respondents who reported a magnitude of  improvement in quality measures and a specific amount saved  attributable to their intervention and (2) the number of respondents  who also reported net cost savings rather than gross cost savings.
To identify key criteria that can be used to assess the strength of  available evidence on the capacity of interventions to enhance the  value of health care we interviewed methodological experts and  conducted a literature review to identify relevant systems for  assessing the strength of evidence. We reviewed methodological  literature published by entities that have well-established systems  for evaluating health care interventions, including the Cochrane  Collaboration; AHRQ’s Effective Health Care Program, which  includes the Evidence-based Practice Centers; and in the United  Kingdom, the National Institute for Health and Clinical Excellence  and the Centre for Reviews and Dissemination. We focused on  those entities with systems for evaluating organizational  interventions that change the structure or delivery of health care.  This led us to pay particular attention to the guidance developed by  Cochrane’s Effective Practice and Organisation of Care (EPOC)  Group, a collaborative review group that specializes in conducting  systematic reviews of organizational interventions.
Our review of this methodological literature and guidance together  with our expert interviews led us to develop a set of questions to  help decision makers and policy analysts who support them to  critically examine the strengths and limitations of evidence about  health care interventions that seek to enhance value. These  questions target three broad areas: (1) assessing the true effect of  the intervention on quality of care and costs, (2) assessing the  scope of study results, and (3) assessing an intervention’s capacity  for replication. We submitted our initial draft questions to several  different experts in assessing the comparative effectiveness of  health care interventions and received their feedback on the  content and clarity of those questions. Based on that feedback, we  made revisions, resulting in the criteria described in our report and  the set of questions listed in appendix IV.
As part of our efforts to identify key criteria for assessing the  strengths and limitations of available evidence on the capacity of  interventions to enhance the value of health care, we examined the  choice of study design used by evaluators to study the interventions  for which we received usable responses to our questionnaire. To  determine the type of study design used to assess the effect of  interventions, we reviewed source documents and questionnaire  responses. (See app. III for more information on study designs.)  Some interventions reported results from multiple studies. In these  cases, we identified each type of study design used to assess the  intervention. We used this information to find the number of  interventions that were assessed using more rigorous study  designs such as randomized controlled trials and the number of  interventions that were assessed using less rigorous study designs  such as pre/post or cross sectional studies.
Our approach is designed to assist decision makers and policy  analysts in assessing the strengths and limitations of evidence  provided to them about the effects of health care interventions on  quality of care and costs. Our approach does not involve the  performance of systematic reviews that could synthesize  information about those effects from multiple studies. Nor does it  attempt to describe a process for producing a numerical or  qualitative rating of the methodological strength of a study along  one or more specified dimensions. Rather, our approach  emphasizes the questions that decision makers and policy analysts  should ask and leaves open the format and content of the answers  to those questions.
To examine factors that can facilitate the implementation and  replication of health care interventions that seek to enhance value,  we analyzed data collected from respondents through our  questionnaire. We reviewed key literature sources and interviewed  experts to identify seven factors that may affect implementation  including leadership support, organizational culture, and  resources. Respondents were asked to indicate, from the list of  close-ended categorical options, to what degree each of the seven  factors facilitated or impeded implementation and to provide an  open-ended explanation of how the factors facilitated or impeded  implementation. We asked respondents who were familiar with the  replication of their intervention to explain if and how the factors  differed from site to site. Respondents were also asked to indicate  the expected degree of importance that each factor could have in  attempting to replicate the intervention as widely as possible and to  explain why these factors were expected to be important. In  addition to the factors identified through our literature review, we  asked respondents to identify and describe up to three additional  factors that facilitated or impeded implementation of their  intervention or that would be important for wide-scale replication. All  close-ended responses were analyzed by assessing the frequency  distribution of responses for each factor. We conducted a content  analysis on open-ended responses to identify common  explanations of how these factors affected implementation and why  these factors would be important for widespread replication of the  intervention.
As part of our analysis of factors that may affect implementation  and replication, we examined differences in questionnaire  responses by the intervention type. To determine the types of  interventions for which we received usable questionnaire  responses, we reviewed source documents and questionnaire  responses for each intervention and assigned them to one of eight  categories (see app. II for more information about intervention  type). To categorize interventions by type we assessed key  intervention characteristics, including the population targeted for  behavior change and levers or activities used to change the way  health care services are delivered. For example, a hospital surgical  team that implemented a checklist was categorized as a patient  safety improvement intervention. Some interventions exhibited key  characteristics of more than one type of intervention. For example,  a primary care practice that implemented a nurse case manager to  facilitate care transitions and employ disease management  strategies exhibits key characteristics of both care coordination or  transition of care programs and chronic condition management  interventions. Interventions that exhibited key characteristics of  more than one type of intervention were categorized in all  appropriate types. To determine if the effect or expected degree of  importance of the factors differed by the type of intervention, we  assessed the frequency distribution of responses for each factor  across intervention type.
Although our efforts to identify relevant interventions for our study  were extensive, we could not ensure that every intervention  meeting our selection criteria had been identified. Therefore the  results from our questionnaire are limited in scope to the 127  interventions for which we received usable responses, and cannot  be generalized to all value-enhancing health care interventions.
section_title: Appendix II: Types of Health Care Interventions That Seek to Improve the Value of Health Care
subsections: section_title: Intervention type (number in study)
paragraphs: Interventions that seek to alter provider  behavior by systematically changing the  basis for provider payments.
Interventions that seek to alter patient  behavior by restructuring health insurance  plan provisions or related health care  benefits.
Interventions that seek to improve care for  patients with chronic conditions.    Can be implemented in either inpatient or  outpatient settings.    Can focus on patient or clinician activities,  Interventions that seek to prevent or reduce  adverse events caused by medical care.  Adverse events include improper  prescriptions or administration of  medications, health-care associated  infections, and pressure sores.
Interventions that facilitate patient transfers  from one setting to another.  Some focus on coordination of patient care  provided by multiple providers.
Providing a single payment, or bundled payment, for  all health care services that are delivered for a defined  episode of care or a specified period of time.  Providing physician group practices performance  payments if the practice meets or exceeds  performance targets.  Insurers offer enrollees a tiered network of providers.  Enrollees who choose a provider in the higher cost  tier pay higher premiums or cost sharing than  enrollees who choose a provider in a lower cost tier.  Enrollees are charged a lower or no copay for specific  drugs that are part of a recommended medical  regimen for a medical condition.  A nurse-social worker team is introduced into a  primary care practice to provide education, help  patients improve self management skills, and develop  care plans with patients.  A multidisciplinary team holds classes for children  with severe asthma and their parents to address  physical needs and group, individual and family  therapy for psychological needs.  A surgical team implements a check list that  enhances team communication and situational  awareness among clinicians to prevent wrong-site  surgeries.  A program of patient risk assessments, specialist  consultations, and new equipment is designed to  minimize pressure sores.  An advanced practice nurse and a trained elder peer  provide support to older adults who are discharged  home after a heart attack or undergoing bypass  surgery to encourage compliance with medications  and lifestyle changes.  A team of nurses and social workers work with  patients with multiple chronic conditions to coordinate  care from multiple providers and to provide ongoing  monitoring and referrals.  A hospital created teams trained in “lean” principles,  based on Toyota’s manufacturing approach, to  identify where changes in routine procedures could  reduce waste and increase efficiency.
Interventions that seek to change health  care organization as a whole through  ongoing and iterative reassessment of  health care practices.  Such interventions seek to both reduce  inefficiency or waste and improve patient  outcomes.
section_title: Intervention type (number in study)
paragraphs: The primary goal is to improve health by  forestalling the development of illness in the  first place.
Programs to promote wellness activities and health  screenings or to prevent falls.  These interventions do not include programs to  prevent adverse events.
Interventions that seek to ensure that  clinical staff adhere to specified treatment  protocols or other forms of standardized  practices.  These interventions seek to modify care  processes by changing where care is  delivered, how care is organized or  structured, or who delivers care.   Multi-site intensive care unit telemedicine program.
A team of clinicians use a four-step mobility protocol  to regularly assess the functional and clinical status of  intensive care unit patients with respiratory failure.
section_title: Appendix III: What Makes Some Study Designs More Rigorous Than Others
paragraphs: The methodological literature on assessing the effect of  interventions places a major emphasis on study design for  identifying those studies that have the capacity to assess an  intervention’s effect on an outcome. The key strength of rigorous  study designs is that they can take account of other factors that  could affect the outcome of interest, and thereby isolate the effect  of the intervention itself.
Randomized controlled trials (RCTs) are widely considered to be  among the most rigorous types of study designs because their  basic structure inherently minimizes the potential impact of  confounding factors on their results. RCTs accomplish this by  randomly allocating study participants to groups that either receive  the intervention—generally referred to as intervention or treatment  groups—or do not receive the intervention—the control groups. The  consequence of random allocation is that the only systematic  difference between study participants in the two groups is exposure  to the intervention. Thus, the effect of all other factors is the same  on the two groups and therefore neutralized in making comparisons  between the intervention and control groups.
A second design type, known as the controlled before and after  study, can be used in situations where the random allocation of  study participants between intervention and control groups required  for an RCT is not feasible. Controlled before and after studies use  data collected from separate treatment and control groups, both  before and after the intervention’s implementation, to help to  separate the effect of the intervention from that of other factors at  work over that time period. In this design type, the control group is  generally chosen in a way that is likely to produce a group that is  broadly similar to the treatment group prior to the implementation of  the intervention. However, methodologists generally recommend an  explicit analysis to compare the intervention and control groups  used in controlled before and after studies in order to demonstrate  that they were in fact similar before the intervention took place.
A third design type, an interrupted time series study, is not based  on a comparison of intervention and control groups. Instead, it  tracks an outcome of interest over time with measurements taken  at many different time points both before and after the intervention.  The multiple data points from before the implementation of the  intervention enable analysts to take account of the impact of other  factors on the outcome and thereby isolate the intervention’s effect  on that outcome. The interrupted time series design works best  when there are data from a substantial number of different time  points, both before and after implementation of the intervention.
Other types of study designs cannot isolate the effect of an  intervention from that of other factors because they provide no  separate information on what would have happened without the  intervention. For example, in a simple pre/post study all one has is  a measurement of the outcome before implementation of the  intervention and a measurement of the outcome after the  intervention. The observed difference reflects all the factors  (including the intervention) affecting the outcome over that time  period. Because confounding factors could potentially affect the  outcome in either the same or the opposite direction as the  intervention, the actual effect of the intervention itself could be  either greater or smaller than the simple pre/post difference. Even  the direction of the intervention’s effect, to increase or decrease the  outcome, could be the opposite of the overall change from pre to  post. That is why the results of a pre/post study generally cannot be  relied on to provide even an approximation of what the likely effect  of a health care intervention is on quality of care and costs.
section_title: Appendix IV: Key Questions for Assessing Evidence from Studies of Interventions That Seek to Enhance Value
paragraphs: The following three tables provide a set of questions that are  intended to help policymakers and others find the information  needed to assess the strengths and limitations of evidence drawn  from studies of health care interventions that seek to enhance value  relating to their impact on quality of care and costs. The three  tables focus on the three broad dimensions described in the body  of this report: (1) the credibility of evidence that attributes changes  in quality of care and costs to the intervention, (2) the applicability  of study results for broader populations of interest, and (3) the  intervention’s capacity for widespread replication.
Each table lists a series of questions that highlight key information  for assessing the evidence produced by relevant studies along with  guidance on how to look for that information in published reports.  Answers to most of these questions may be found in relevant  sections of those reports; if not, one can ask the investigators who  conducted the studies. While this set of questions is selective and  does not cover every potential methodological issue, the  information it calls for should provide policymakers a basis for  making an informed assessment of the overall credibility and scope  of the available evidence regarding the apparent impact of these  interventions on quality of care and costs, as well as the  demonstrated capacity of those interventions for widespread  replication.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the individual named above, Jessica Farb, Assistant  Director; Kristin Ekelund; Krister Friday; Katie Mack; and   Eric Peterson made key contributions to this report.