Value in Health Care: Key Information for Policymakers to Assess Efforts to Improve Quality While Reducing Costs
Released: 2011-07-26T00:00:00

Letter

Background

The methodological literature provides insight into conducting systematic assessments of evidence for health care interventions that change the delivery or structure of care. Furthermore, the literature on organizational change is pertinent to understanding the key factors that can facilitate or impede implementation and replication of such health care interventions.

Applied social science research has developed a core set of methodological questions and approaches for assessing the effect of programs or other interventions on a wide range of organized activities. They address two key issues: how best to determine the independent effect of a program or intervention and how best to generalize from the results obtained from one or more studies to broader populations of interest. A number of organizations have developed more specialized guidance for applying these general methodological principles to health care interventions. For example, the Effective Practice and Organisation of Care Group (EPOC) is a component of the Cochrane Collaboration—an international network of individuals who analyze the effect of health care interventions—which focuses on interventions that change the practice of care and the delivery of health care services. EPOC provides guidance to researchers on how best to prepare systematic reviews of such interventions in order to synthesize the information available in multiple studies. AHRQ’s Effective Health Care Program (EHC) and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group have developed similar guidance, though both of these efforts focus more on assessing alternative medical treatments rather than alternative approaches for organizing health care services.

Research on organizational change has identified certain factors as key contributors to successful implementation of health care interventions for quality improvement. For example, the literature consistently cites leadership support as essential for successful implementation of quality-of-care interventions within health care settings. Further, it describes the role of leaders in promoting the adoption of interventions by their organizations, in winning acceptance among affected staff members for the changes those interventions entail, and in marshalling sufficient resources for the intervention to succeed. In addition, this literature has shown that organizations vary in their attitudes, beliefs, and values, and that this “organizational culture” can either promote or inhibit change. Organizations tend to achieve quality of care improvement more readily if they have a culture with such characteristics as receptiveness to change, placing high value on ensuring the quality of care provided, and prizing innovation with a willingness to take risks. The literature also cites the role of infrastructure factors, such as the sufficiency and appropriateness of staff resources and the adequacy of existing health information technology (health IT) systems, in the successful implementation of quality improvement interventions. Another factor cited in the literature is the availability of previously developed tools and procedures for standardizing health care processes—such as checklists or guidelines—as well as other types of technical assistance that can facilitate the implementation of a given intervention. Additionally, the literature has pointed to financial factors that affect the implementation of interventions for quality improvement, including both the level of financial resources needed to sustain an intervention and the use of financial incentives to promote quality enhancement activities. Financial incentives represent a particular application of financial resources that involve the contractual or other provisions that determine how much health care providers are paid and for what. Such financial incentives affect who benefits from and who pays for the cost of an intervention. This in turn can facilitate or impede the implementation and replication of interventions.

Respondents Reported Basic Information Needed to Assess Value Available for About Half of Selected Interventions

About half of the respondents to our questionnaire reported basic information on the effect of their intervention on both quality of care and costs—the two types of data needed to determine whether or to what extent a particular intervention enhanced the value of health care. Overall, the vast majority of our respondents reported at least some information on the observed effect of their intervention on quality of care. Relatively fewer—though still over half—of our respondents reported at least some information on the effect of their intervention on costs.

The ability of policymakers to identify interventions that substantially improve quality and reduce costs depends on the availability of basic information on the size of the effect of an intervention on both quality of care and costs. These are the two types of data needed to determine whether or to what extent a particular intervention enhanced the value of health care. Just over half of the respondents to our questionnaire reported such basic information on their interventions. Sixty-four of 127 respondents reported information on both improvements observed in at least one quality measure and a specific amount of cost savings (see table 1). For the remaining interventions, the missing information most often concerned the effect of the intervention on costs. Furthermore, even fewer respondents, 45, reported improvements observed in at least one quality measure and a specific amount of cost savings that accounted for the costs of implementing their intervention—net cost savings.

Compared to information on both quality of care and costs, information on the effect of selected interventions on quality of care alone was more frequently reported. The vast majority of respondents to our questionnaire reported at least some information on the observed effect of their intervention on quality of care. Specifically, 114 of 127 respondents reported improvements in one or more measures used to assess the effect of their intervention on various aspects of care quality. Of these, 112 respondents reported a specific magnitude of improvement observed in at least one quality measure in terms of a percentage change or other quantitative measurement. Additionally, 2 respondents reported improvement in at least one quality measure, but did not report a specific magnitude of improvement. In contrast, the remaining 13 respondents did not report sufficient information to determine whether their intervention had any effect on quality of care. Six of 127 respondents described one or more measures used to assess the effect of their intervention on different aspects of care quality, but did not report a magnitude of improvement observed in these measures. Seven respondents did not report any information on the measures used to assess the effect of their intervention on aspects of care quality.

Respondents reported that the effect of their intervention on quality of care was assessed using a range of measures that generally fell into five broad types reflecting different aspects of care quality (see table 2). Respondents most frequently described one or more quality measures that were used to assess the effect of their intervention on outcomes resulting from care. Specifically, 82 respondents reported that the effect of their intervention on quality of care was assessed using outcome measures such as patient mortality, the overall physical and emotional health of a patient, or the level of stress reported by a patient caregiver. In addition, 56 respondents described one or more measures that assessed the effect of their intervention on the amount of health care services consumed. These measures included the length of hospital stay, the number of emergency department visits, and the number of hospital readmissions for a specified population. Forty- four respondents described measures that assessed the effect of their intervention on processes of care. Process-of-care measures assess the extent to which the care provided to a patient was appropriate based on current professional knowledge and the particular circumstances. For example, process-of-care measures could examine whether diabetes patients had received foot exams, eye exams, and regular glucose monitoring at specified intervals.

Fewer respondents described measures that assessed quality in terms of the experience of a patient or caregiver or the structure of care.

Although the information provided by any one type of quality measure is limited, most of our respondents reported that the effect of their intervention on quality of care was assessed using more than one type of quality measure. Each type of quality measure offers insight into a particular domain of quality such as outcomes of care, processes of care, or experience of care. Just 41 respondents reported that only one type of measure was used to assess the effect of their intervention on quality of care (see fig. 1). For most—79—of the interventions in our review, respondents reported that the effect of their intervention on quality of care was assessed using measures belonging to two or more different types of quality measures, thereby providing a broader perspective on the effect of the intervention on quality of care.

Somewhat fewer respondents to our questionnaire reported information on the effect of their interventions on costs than quality of care. Specifically, 72 of 127 respondents reported a specific amount of change in costs—cost savings. Respondents most frequently reported that costs were assessed by calculating the total dollars saved or the average dollars saved per person annually. Respondents less frequently reported that costs were assessed by calculating the financial return on investment, percentage change in total health care costs per patient, or an alternative cost metric such as dollars saved per member per month for patients participating in a certain health care plan. In contrast, the remaining 55 respondents did not report sufficient information to determine whether their intervention had any effect on costs. Nine of 127 respondents reported that costs were assessed, but did not report a specific amount of cost savings. Forty-five respondents reported that cost savings were not assessed, and one respondent did not report any information on whether cost savings were assessed.

Most, but not all, of the respondents who reported a specific amount of cost savings stated that these cost savings accounted for the costs associated with implementing the intervention. Among the 72 respondents who reported a specific amount of cost savings, 51 respondents reported net cost savings that took account of implementation costs; another 20 respondents reported gross cost savings that did not take implementation costs into account. When asked to provide additional detail on their implementation cost calculations, 35 respondents reported that the cost savings took account of both start-up costs associated with developing and initially implementing the intervention as well as ongoing costs associated with operating and maintaining the intervention over time. Two respondents reported that cost savings took account of start-up costs but not ongoing costs to maintain the intervention, and 19 reported that cost savings took account of ongoing costs but not start-up costs.

The interventions we reviewed also varied in the extent to which the reported cost savings attributed to them were based on information directly related to the intervention. Forty-nine respondents reported that cost savings were calculated using only data that were collected specifically to assess the effect of their intervention on costs. In contrast, 26 respondents reported that cost savings were calculated using a mix of data that were collected specifically to assess the intervention and data from a secondary source such as published literature or a national database. For example, one respondent reported cost savings attributable to an intervention designed to improve patient self-management of asthma based on data that were collected on changes over time in the actual number of health care encounters for patients enrolled in the program and the estimated costs for those encounters derived from national averages for several types of health care services such as hospital days or emergency department visits. While data from secondary data sources may provide otherwise missing information needed to estimate the cost savings achieved by an intervention, the relevance of such secondary data to that particular intervention may be open to question, which makes the accuracy of the cost savings estimate more uncertain.

The Strength of Evidence on the Effect of Interventions Can Be Assessed along Three Dimensions

Policymakers and others can assess the strength and limitations of available evidence from studies on the effect of health care interventions on quality of care and costs along three dimensions. One, the credibility of evidence on the effect of health care interventions on quality of care and costs depends primarily on whether those studies apply rigorous study designs. Two, the applicability of the results of studies to a broader population depends on the extent to which the study population is representative of that larger population. Finally, the capacity of health care interventions for widespread replication can be examined in terms of the consistency of results obtained by each intervention across diverse organizations. Appendix III provides a more detailed explanation of what makes some study-design types more rigorous than others and appendix IV presents a list of key questions that describe the information that policymakers can look for to assess the evidence provided by particular studies along these three dimensions.

For policymakers and others, the benefit obtained from basic information on the effect of interventions on quality of care and costs depends in large part on the strength of that evidence. Information based on weak evidence can provide policymakers a misleading indication of an intervention’s potential to enhance value. For example, the direction and magnitude of the changes in quality of care and cost reported for the 127 interventions examined through our questionnaire could deviate substantially from the actual impact of those interventions, depending on the characteristics of the studies that generated that reported information. To determine what information has the kind of evidentiary support that they can rely on, policymakers can assess the strengths and limitations of studies that examine health care interventions of interest along three broad dimensions.

The first of these dimensions is the credibility of evidence that attributes any changes in quality of care and costs to those interventions. The methodological experts we consulted uniformly emphasized the primacy of study design in determining the credibility of evidence on the effect of health care interventions on quality of care and costs. Observed changes in quality of care and costs that one might attribute to a health care intervention may in fact be due in large part to the effect of a wide variety of other factors. The choice of study design type is critical because rigorous designs have the capacity to isolate the effects of a health care intervention from other factors that may affect changes in quality of care and costs.

The methodological literature we reviewed identifies several different study design types that have sufficient rigor to isolate the effect of interventions on quality of care and costs. They include randomized controlled trials (RCTs), interrupted time series studies, and controlled before and after studies. RCTs and controlled before and after studies both use control groups—consisting of study participants who are not exposed to the intervention—to adjust for the effect of other factors besides the intervention. Interrupted times series studies do not use control groups; instead they rely on analyzing data collected at multiple time points both before and after an intervention is implemented to adjust for other factors. (See app. III for more information on how these study design types isolate the effect of an intervention.)

In contrast, according to the methodological literature we reviewed, some other types of study designs lack the capacity to isolate the effect of a health care intervention from that of other factors. For example, a simple pre/post study that assesses quality of care and costs once before an intervention is implemented and a second time after implementation of the intervention has no mechanism analogous to a control group to take account of the effect of other factors. The same is true for post-only studies that rely entirely on data collected after an intervention was implemented. With studies using these types of designs, there is no way to determine how much of the difference observed between the pre and post measurements, or among any groups following an intervention, was due to the intervention and not to other factors. Consequently, such studies will not provide policymakers credible information about the extent to which the intervention itself affected both quality of care and costs.

Table 3 describes key distinguishing characteristics to help policymakers identify the type of study design employed in a study of an intervention.

Among studies addressing the effect of health care interventions on quality of care and costs, a range of rigorous to weak design types are used. For example, among the 127 interventions for which we received responses to our questionnaire, we found 22 interventions with studies involving RCTs and another 11 interventions assessed using controlled before and after studies. However, for a substantially larger number of the 127 interventions, the studies we identified employed the types of study designs that do not isolate the effect of the intervention from other factors. Specifically, the results for 67 interventions were based on pre/post studies, and another 19 were based on post-only studies of one kind or another. In this one, diverse set of interventions that we reviewed, policymakers could find credible evidence based on rigorous study designs concerning the effects of certain interventions on quality of care and costs; however, for many other interventions such studies were lacking.

In addition to study design, the methodological literature we reviewed emphasized the importance of how a study is conducted. Even rigorous study designs can lose their capacity to isolate the effect of an intervention on quality of care and costs if researchers do not adhere to the requirements of those designs. Thus, assessments of the strengths of study results should consider how well the study design was implemented.

One component of a study’s implementation that policymakers can examine involves the selection and management of control groups used in the study. In order to isolate the effects of an intervention, the control group has to be equivalent to the treatment group— except for the latter’s exposure to the intervention. According to the methodological literature we reviewed, that equivalence can be compromised in a number of ways. In the case of RCTs, for example, allocation to treatment and control groups may not be truly random if there are flaws in the process for assigning study subjects to those groups. Moreover, for both RCTs and controlled before and after studies, losing a disproportionate number of study participants from either treatment or control groups can also undermine their equivalence.

Another component of a study’s implementation that policymakers can examine concerns the measures and procedures adopted for data collection. According to the methodological literature we reviewed, a study will produce stronger evidence when it employs measures that are recognized as valid and reliable. For example, central line-associated bloodstream infections can be tracked using a surveillance measure developed by the Centers for Disease Control (CDC) or with less labor-intensive measures that draw on administrative data. Clinicians consider the CDC measure to be the most valid and reliable measure for this type of infection because it calls for laboratory confirmation of identified infections and it accounts for varying risks of infection based on the number of days that a central line catheter is in place. In addition, the data for those measures should be collected at the same time and in the same way from all groups in the study. Any systematic inconsistencies in how data are collected for a study can skew the results.

If a study produces credible evidence that a health care intervention has a positive effect on both quality of care and costs within the population it examined, a second dimension that policymakers and others can assess concerns the scope of that effect—for what broader populations or groups are the results applicable? Applicability depends on the representativeness of the study population for a broader population of interest. The methodological literature identifies two different approaches for establishing representativeness: (1) randomly selecting the study population from a known universe, or (2) examining the degree to which a study population matches a given broader population on characteristics relevant to the intervention. The first approach, random selection, intrinsically makes the study population representative of the particular universe from which it was selected and the study results applicable to that population.

The second approach for establishing representativeness— examining the extent of similarity between the study population and a broader population of interest—can be used by policymakers whenever the study population was not chosen randomly or the broader population of interest to policymakers is not the universe from which the study population was selected. Policymakers can assess the degree of similarity between the study population and a broader population through an examination that focuses on two issues: (1) identifying characteristics where the study population and broader population of interest differ and (2) assessing whether any differences found could influence the effect of the intervention on quality of care and costs (see app. IV ).

Major differences between a nonrandomly selected study population and a broader population of interest to policymakers should raise questions about the applicability of the study’s results for that broader population. For example, an intervention to improve care coordination for patients with diabetes might be implemented and assessed in a few academic medical centers. In that situation, the representativeness of the study population for all patients with diabetes could come into question on at least two counts—the kind of care provided in an academic medical center might well differ from that usually provided by community-based providers and the patients treated by academic medical centers might have a higher level of severity than diabetics treated elsewhere. If patients in the study received a different overall set of services, that could affect the impact of the intervention on those patients even if the intervention itself were implemented the same way for the two populations. Similarly, an intervention could have a more pronounced effect on patients with a higher level of severity, or the intervention might work less well for such patients. Thus, to establish the applicability of the study results to a broader population of diabetic patients, studies of the intervention would need to provide evidence that the differences between the study population and the broader population of diabetics would not affect the performance of the intervention.

A third dimension on which policymakers and others can assess the strength of evidence for health care interventions concerns the capacity of an intervention for replication across diverse organizations. Because organizations vary across the factors that affect the implementation of health care interventions, including leadership, organizational culture, and staff and financial resources, a particular intervention may work more or less well depending on the organizational environment in which it operates. As a result, some organizations may be more receptive to a particular value- enhancing intervention than others. That, in turn, can make it more difficult to take an intervention that proved successful in a small number of organizations and replicate it widely across many others. However, some interventions have produced positive results on quality of care and costs in a range of different organizations, which suggests that they may be less sensitive to varying circumstances across organizations.

According to the methodological literature and experts that we consulted, certain information can provide the basis for an assessment of the consistency in an intervention’s effects on quality of care and costs in different organizations. Specifically, this information concerns the number of different organizations where the intervention has been implemented, the degree of diversity exhibited by those organizations, and the consistency in observed changes in quality of care and costs across those organizations. However, such information would not be available for assessing the consistency of results across diverse organizations if an intervention has been implemented in only a few different organizations, or in multiple organizations that are generally quite similar. That is also the case if studies only analyze and report changes in quality of care and costs attributed to an intervention in the aggregate, rather than separately for the different organizations that implemented it.

On the other hand, for interventions that have been implemented in multiple, diverse organizations, and their results analyzed separately at the different organizations, it is possible for policymakers to compare the results of the intervention across those organizations to examine the consistency of the intervention’s effect. To the extent that those interventions consistently produce positive effects on quality of care and costs among diverse organizations, that provides evidence of their capacity for widespread replication. For other interventions, if data on the changes in quality of care and costs across the different organizations indicate a lack of consistency in outcomes, that provides evidence of a more restricted capacity for replication.

Leadership Support and Other Factors Reported As Important for Both Implementation and Replication of Interventions

Respondents to our questionnaire reported, generally by large margins, that leadership support as well as other factors, such as organizational culture and staff resources, significantly facilitated implementation. However, respondents were more divided when asked about the reported effect that health IT had on implementation, and most respondents reported that financial incentives were not a factor in the implementation of their intervention. A majority of respondents reported that each of these factors, with the exception of financial incentives, was expected to be either very or somewhat important if one were to attempt to replicate their intervention as widely as possible.

Taking account of factors that prior research has shown tend to facilitate or impede the implementation and replication of interventions may enhance efforts by policymakers and others to promote the adoption of interventions across varied organizational contexts. In examining the relative impact of seven factors identified in our literature review, we found that respondents to our questionnaire reported, generally by large margins, that five of the seven factors significantly facilitated implementation of their intervention. Health IT and financial incentives were the exceptions. Leadership support was the factor that the largest number of respondents reported as having significantly facilitated implementation of their intervention (see table 4). When asked to describe how leadership support facilitated implementation, respondents frequently explained that a leader who visibly prioritized and endorsed the intervention, allocated necessary resources, and championed the development and implementation of the intervention and drove necessary organizational or behavioral changes facilitated the implementation of the intervention. Respondents also explained that having champions, specifically clinicians, was a key factor in encouraging cooperation and participation in the intervention by staff, especially fellow clinicians. The prominent role attributed to leadership in implementing the many different types of interventions in our sample suggests that policymakers will have greater success in implementing and replicating interventions to the extent that they can take steps to ensure that strong leadership is in place before interventions are initiated.

Respondents typically reported that a combination of additional factors along with leadership support significantly facilitated implementation of their intervention. The 92 respondents who reported leadership support as having significantly facilitated implementation, reported, on average, another three factors as having significantly facilitated implementation. Of the 86 respondents who reported at least one factor in addition to leadership support as significantly facilitating implementation, more than half reported staff resources (60), organizational culture (55), and the availability of other tools (50), respectively, as having significantly facilitated implementation. Nearly half (42) reported that financial resources, in addition to leadership, significantly facilitated implementation. Just six respondents reported leadership support and no other factor as having significantly facilitated implementation.

In contrast to the five factors that a clear majority of respondents reported having facilitated implementation of their intervention, respondents were more divided on how health IT affected implementation, as shown in table 4. Health IT had the highest number of respondents, compared to the other factors, that reported the factor impeded implementation of their intervention. Further, a substantial group of respondents reported that health IT was not a factor. On the other hand, nearly half of respondents reported that health IT either significantly or somewhat facilitated implementation of their intervention. Respondents frequently explained that health IT facilitated implementation of their intervention by enhancing the exchange of information and communication across providers or organizations, facilitating the collection of data or the evaluation of the intervention and improving the efficiency and productivity of staff. Of those who reported that health IT impeded implementation, respondents commonly cited the limited functional capacity of existing systems or the lack of interoperability across settings as impediments to successful implementation. Other respondents explained that the general lack of health IT altogether acted as a barrier that impeded implementation. Variation in the role of health IT across different types of interventions does not appear to explain the mixed assessment of this factor; as respondents for each of the intervention types included in our sample—with two exceptions— were similarly divided on how health IT affected implementation. However, proportionately more respondents for care coordination or transitions of care interventions as well as care-process- improvement interventions reported health IT as having facilitated implementation compared to respondents for other types of interventions. This result suggests that as policymakers consider different health care interventions, implementation of some of their options will depend more heavily than others on having appropriately configured health IT in place.

Financial incentives were most often reported as not a factor. Slightly more than half of our 127 respondents reported financial incentives—as distinct from the related, but broader, financial resources factor—as having not been a factor in implementation of their intervention. The exception was for the two types of interventions for which financial incentives were an integral component—provider payment restructuring and insurance redesign—where respondents most often reported financial incentives as having significantly facilitated implementation. When asked to explain how financial incentives facilitated or impeded implementation most respondents simply provided a description of the incentives they used to implement the intervention, such as payments to providers or patients to participate in the intervention. However, a few respondents explained that the expected cost savings generated from the intervention was an indirect incentive to implement the intervention while other respondents explained that incentives within existing payment systems, or the lack thereof, affected implementation. While the implementation of many interventions included in our sample may not have been affected by financial incentives, current means of paying for health care, such as fee-for-service payment structures, may have hindered the successful implementation of other interventions.

Much as they had reported regarding the implementation of their intervention, nearly all respondents consistently expected that leadership support would be very important if one were to attempt to replicate their intervention as widely as possible (see table 5). Leadership support was reported nearly unanimously by respondents as being very important for widespread replication of their intervention, paralleling respondents’ relatively consistent assessment of the effect of leadership on implementation. In addition, a clear majority of respondents expected that each of the other factors—except for financial incentives—would be either very or somewhat important for replication.

In contrast to the highly divided views health IT evoked from respondents regarding its role in the implementation of their interventions, it was reported by a substantial majority of respondents as either very (48) or somewhat (48) important for widespread replication. This could be an indication that, if health-IT- related impediments experienced when implementing the intervention, such as the lack of interoperability across settings, were ameliorated, health IT could be important to the successful replication of some interventions. Similar to views expressed about the implementation of care coordination or transitions of care interventions, respondents for these types of interventions commonly reported that health IT would be very important for widespread replication more so than respondents for other types of interventions.

Financial incentives was the factor that drew the most mixed assessment from respondents with regards to its expected importance for the widespread replication of interventions. Nearly half of respondents indicated that financial incentives were not important for widespread replication, which is similar to the view of most respondents regarding the role of such incentives in the implementation of their interventions. Another substantial group of respondents (30) indicated that financial incentives would be very important for replication. When respondents were asked to explain why factors would be important for widespread replication, respondents discussed financial factors more frequently than any other factor. Respondents’ explanations about these financial factors often concerned a misalignment of financial incentives within existing payment systems that limited the attractiveness of replicating interventions that seek to enhance value. For example, some respondents noted that it would be difficult to replicate interventions that involved providing additional services, such as care coordination, under existing payment systems that typically do not compensate providers for those services.

Concluding Observations

Our work suggests that progress in achieving greater value in health care in the U.S. will depend, in part, on the availability of information regarding the effect of different interventions on quality of care and costs and on how policymakers and others assess and use that information. Such information can guide the choices of policymakers among multiple interventions vying for support, but those decisions will have a sounder basis if the information meets certain criteria regarding its content and strength of evidence. With respect to content, information on the magnitude of an intervention’s effect on both quality of care and costs is needed to determine if an intervention has enhanced value. In the case of the responses to our questionnaire on 127 diverse interventions, we found that this basic level of information was reported as available about half the time. With respect to the strength of evidence, the most critical indication comes from the types of study designs used to produce that information. There are a range of rigorous study designs which can provide credible support for the attribution of observed changes in quality of care and costs to a particular intervention. Our review of studies associated with the 127 interventions examined by our questionnaire found that while a number of studies employed rigorous study designs, a substantially larger number employed weaker designs that could not isolate the effect of an intervention from other factors. To the extent that policymakers find and use information on health care interventions that provides sufficient credible evidence on the effects of those interventions on both quality of care and costs, they will be better equipped to determine which interventions produce greater value in health care. Our work also suggests that successful efforts to encourage the widespread adoption of value-enhancing interventions will need to take into account a complex mix of factors, including leadership support, organizational culture, and staff resources, that facilitate the implementation of health care interventions across a wide range of organizational contexts.

Agency Comments

We requested comments from the Department of Health and Human Services, but none were provided.

We are sending copies of this report to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To examine the availability of information on the effect of selected health care interventions on quality of care and costs as well as factors that can facilitate the implementation and replication of these interventions, we studied a diverse set of specific interventions that seek to enhance the value of health care through making changes in the way care is delivered. Specifically, these interventions make changes in who delivers health care services, how care is organized, or where care is delivered for a specified population. To identify interventions for our study, we drew upon six distinct sources to select a broad and diverse, though not exhaustive, set of interventions that have been implemented in one or more locations in the U.S. or abroad. These sources allowed us to identify a wide range of value-enhancing strategies implemented in different of health care settings, such as hospitals, integrated delivery systems, and physician practices, over more than a 10-year time span, including interventions that have not been described in academic or professional literature. We identified 828 interventions of potential relevance to our study through the following six sources:  A review of relevant literature on health care interventions that make changes in who delivers care services, how it is delivered, or where it is delivered. We conducted several searches using online databases, including Medline and ProQuest Health, to identify articles on interventions that were published from 1999 to 2009.  A review of interventions contained in the Agency for Healthcare Research and Quality’s (AHRQ) Health Care Innovations Exchange (HCIE) as of August 20, 2009. The HCIE is a Web site that acts as a repository for information on quality improvement interventions and other innovative strategies to improve health care submitted by their implementers. Many of the interventions contained in the HCIE make changes in the way health care is delivered and include information on cost as well as quality.  A review of the relevant articles contained in the Tufts Medical Center’s Cost-Effectiveness Analysis (CEA) Registry that were published from 1999 to 2009. The CEA Registry is a comprehensive database of health care cost-utility analyses that examine the health benefits and costs of strategies to improve health care. The CEA Registry contains articles from 45 peer- reviewed publications.

Interviews with experts on health care interventions associated with organizations such as state governments, integrated delivery systems, employer groups, and other countries.

Information on interventions that we identified from press reports, select journal articles published after 2009, and presentations at conferences.

Information on interventions submitted by their innovators or evaluators to either the Senate Budget Committee or GAO.

To select interventions for inclusion in our study, we reviewed source documents for each of the potentially relevant interventions that we identified through our six sources. We selected 239 interventions that met the following seven criteria:  The intervention made a discrete change in who delivers health care services, how care is organized, or where care is delivered.  The intervention targeted a population or problem that was relevant to the U.S. health care system.  The intervention may have included health information technology (health IT) as one of its components of change, but health IT was not the intervention’s only component of change.  The primary goal of the intervention was not focused on increasing access to care.  The intervention activities must fall within the health care system.  The source document or documents for the intervention either contained information or indicated that information is available on the effect of the intervention on quality of care and its effect on costs. Moreover, the source documents indicated that the intervention enhanced the value of health care by meeting one of the following three conditions: (1) increases quality of care and reduces costs; (2) maintains quality of care and reduces costs; or (3) increases quality of care and maintains costs.  The intervention was implemented in at least one health care setting. Interventions that were studied by examining their potential costs and benefits based on simulated outcomes rather than analyzing data from their actual implementation were excluded.

To collect information on the 239 health care interventions that we selected for our study, we developed a Web-based questionnaire that contained 22 open- and closed-ended questions on interventions, their effect on quality of care and costs, and factors that may affect their implementation and replication. We sent our questionnaire to 235 individuals who participated in developing, implementing, or evaluating each intervention. We identified these individuals through the source documents that we used to select interventions for our study. We received usable responses— responses that contained relevant information on the effect of the intervention on quality of care, the effect of the intervention on costs, or key factors that may affect implementation—for 127 interventions. We developed protocols for cleaning and analyzing data that we received from questionnaire respondents. These protocols included: identifying usable responses; reviewing source documents to clarify responses; and, if necessary, contacting respondents directly to obtain additional information on their intervention.

To determine the availability of information on the effect of selected health care interventions on quality of care, we analyzed data that we collected from respondents through our questionnaire. We asked respondents to describe up to five key measures used to assess the effect of their intervention on quality of care and the magnitude—a percentage change or other quantitative assessment—of change observed in each measure described relative to a control group that did not experience the intervention or a baseline assessment made prior to implementing the intervention. We conducted a content analysis on questionnaire responses to determine the number of respondents who described one or more key measures used to assess the effect of their intervention on quality of care and the number of respondents who reported improvements in those measures attributable to their intervention.

As part of our analysis on the availability of information on the effect of selected health care interventions on quality of care, we examined the types of quality measures respondents reported. We conducted a content analysis on questionnaire responses to determine what aspect of care quality—such as patient mortality, hospital readmissions, or patient satisfaction with care—each measure examined. We categorized each measure that respondents described by type based on the aspect of care quality it examined; for example, we categorized a measure that assessed the effect of an intervention on patient mortality as an outcome measure. We categorized quality measures into types that are largely based on the measure domains laid out by AHRQ in its National Quality Measure Clearing House. We did not include all measure domains laid out by AHRQ in our analysis, because some domains, such as access to care, fell outside of the scope of our engagement. Moreover, measures that did not clearly specify which aspect of care quality was assessed were categorized as unspecified measures. We analyzed this information to determine the types of quality measures used to assess the effect of each intervention on quality of care.

To determine the availability of information on the effect of selected health care interventions on costs, we analyzed data that we collected from respondents through our questionnaire. We asked respondents to report the type of cost savings, such as total dollars saved or dollars saved per patient, calculated to assess the effect of their intervention on costs and the specific amount saved for type of cost savings calculated. We also asked respondents if their reported savings accounted for costs associated with implementing the intervention and what information was used to calculate those savings. We determined the number of respondents who reported calculating each type of savings and a specific amount saved for those savings. Furthermore, we analyzed responses by finding the number of respondents who reported accounting for costs associated with implementing the intervention—net cost savings— and the type of information used to calculate those savings. Additionally, we used this information along with information we obtained through our analysis of quality measures to determine (1) the number of respondents who reported a magnitude of improvement in quality measures and a specific amount saved attributable to their intervention and (2) the number of respondents who also reported net cost savings rather than gross cost savings.

To identify key criteria that can be used to assess the strength of available evidence on the capacity of interventions to enhance the value of health care we interviewed methodological experts and conducted a literature review to identify relevant systems for assessing the strength of evidence. We reviewed methodological literature published by entities that have well-established systems for evaluating health care interventions, including the Cochrane Collaboration; AHRQ’s Effective Health Care Program, which includes the Evidence-based Practice Centers; and in the United Kingdom, the National Institute for Health and Clinical Excellence and the Centre for Reviews and Dissemination. We focused on those entities with systems for evaluating organizational interventions that change the structure or delivery of health care. This led us to pay particular attention to the guidance developed by Cochrane’s Effective Practice and Organisation of Care (EPOC) Group, a collaborative review group that specializes in conducting systematic reviews of organizational interventions.

Our review of this methodological literature and guidance together with our expert interviews led us to develop a set of questions to help decision makers and policy analysts who support them to critically examine the strengths and limitations of evidence about health care interventions that seek to enhance value. These questions target three broad areas: (1) assessing the true effect of the intervention on quality of care and costs, (2) assessing the scope of study results, and (3) assessing an intervention’s capacity for replication. We submitted our initial draft questions to several different experts in assessing the comparative effectiveness of health care interventions and received their feedback on the content and clarity of those questions. Based on that feedback, we made revisions, resulting in the criteria described in our report and the set of questions listed in appendix IV.

As part of our efforts to identify key criteria for assessing the strengths and limitations of available evidence on the capacity of interventions to enhance the value of health care, we examined the choice of study design used by evaluators to study the interventions for which we received usable responses to our questionnaire. To determine the type of study design used to assess the effect of interventions, we reviewed source documents and questionnaire responses. (See app. III for more information on study designs.) Some interventions reported results from multiple studies. In these cases, we identified each type of study design used to assess the intervention. We used this information to find the number of interventions that were assessed using more rigorous study designs such as randomized controlled trials and the number of interventions that were assessed using less rigorous study designs such as pre/post or cross sectional studies.

Our approach is designed to assist decision makers and policy analysts in assessing the strengths and limitations of evidence provided to them about the effects of health care interventions on quality of care and costs. Our approach does not involve the performance of systematic reviews that could synthesize information about those effects from multiple studies. Nor does it attempt to describe a process for producing a numerical or qualitative rating of the methodological strength of a study along one or more specified dimensions. Rather, our approach emphasizes the questions that decision makers and policy analysts should ask and leaves open the format and content of the answers to those questions.

To examine factors that can facilitate the implementation and replication of health care interventions that seek to enhance value, we analyzed data collected from respondents through our questionnaire. We reviewed key literature sources and interviewed experts to identify seven factors that may affect implementation including leadership support, organizational culture, and resources. Respondents were asked to indicate, from the list of close-ended categorical options, to what degree each of the seven factors facilitated or impeded implementation and to provide an open-ended explanation of how the factors facilitated or impeded implementation. We asked respondents who were familiar with the replication of their intervention to explain if and how the factors differed from site to site. Respondents were also asked to indicate the expected degree of importance that each factor could have in attempting to replicate the intervention as widely as possible and to explain why these factors were expected to be important. In addition to the factors identified through our literature review, we asked respondents to identify and describe up to three additional factors that facilitated or impeded implementation of their intervention or that would be important for wide-scale replication. All close-ended responses were analyzed by assessing the frequency distribution of responses for each factor. We conducted a content analysis on open-ended responses to identify common explanations of how these factors affected implementation and why these factors would be important for widespread replication of the intervention.

As part of our analysis of factors that may affect implementation and replication, we examined differences in questionnaire responses by the intervention type. To determine the types of interventions for which we received usable questionnaire responses, we reviewed source documents and questionnaire responses for each intervention and assigned them to one of eight categories (see app. II for more information about intervention type). To categorize interventions by type we assessed key intervention characteristics, including the population targeted for behavior change and levers or activities used to change the way health care services are delivered. For example, a hospital surgical team that implemented a checklist was categorized as a patient safety improvement intervention. Some interventions exhibited key characteristics of more than one type of intervention. For example, a primary care practice that implemented a nurse case manager to facilitate care transitions and employ disease management strategies exhibits key characteristics of both care coordination or transition of care programs and chronic condition management interventions. Interventions that exhibited key characteristics of more than one type of intervention were categorized in all appropriate types. To determine if the effect or expected degree of importance of the factors differed by the type of intervention, we assessed the frequency distribution of responses for each factor across intervention type.

Although our efforts to identify relevant interventions for our study were extensive, we could not ensure that every intervention meeting our selection criteria had been identified. Therefore the results from our questionnaire are limited in scope to the 127 interventions for which we received usable responses, and cannot be generalized to all value-enhancing health care interventions.

Appendix II: Types of Health Care Interventions That Seek to Improve the Value of Health Care

Intervention type (number in study)

Interventions that seek to alter provider behavior by systematically changing the basis for provider payments.

Interventions that seek to alter patient behavior by restructuring health insurance plan provisions or related health care benefits.

Interventions that seek to improve care for patients with chronic conditions.  Can be implemented in either inpatient or outpatient settings.  Can focus on patient or clinician activities, Interventions that seek to prevent or reduce adverse events caused by medical care. Adverse events include improper prescriptions or administration of medications, health-care associated infections, and pressure sores.

Interventions that facilitate patient transfers from one setting to another. Some focus on coordination of patient care provided by multiple providers.

Providing a single payment, or bundled payment, for all health care services that are delivered for a defined episode of care or a specified period of time. Providing physician group practices performance payments if the practice meets or exceeds performance targets. Insurers offer enrollees a tiered network of providers. Enrollees who choose a provider in the higher cost tier pay higher premiums or cost sharing than enrollees who choose a provider in a lower cost tier. Enrollees are charged a lower or no copay for specific drugs that are part of a recommended medical regimen for a medical condition. A nurse-social worker team is introduced into a primary care practice to provide education, help patients improve self management skills, and develop care plans with patients. A multidisciplinary team holds classes for children with severe asthma and their parents to address physical needs and group, individual and family therapy for psychological needs. A surgical team implements a check list that enhances team communication and situational awareness among clinicians to prevent wrong-site surgeries. A program of patient risk assessments, specialist consultations, and new equipment is designed to minimize pressure sores. An advanced practice nurse and a trained elder peer provide support to older adults who are discharged home after a heart attack or undergoing bypass surgery to encourage compliance with medications and lifestyle changes. A team of nurses and social workers work with patients with multiple chronic conditions to coordinate care from multiple providers and to provide ongoing monitoring and referrals. A hospital created teams trained in “lean” principles, based on Toyota’s manufacturing approach, to identify where changes in routine procedures could reduce waste and increase efficiency.

Interventions that seek to change health care organization as a whole through ongoing and iterative reassessment of health care practices. Such interventions seek to both reduce inefficiency or waste and improve patient outcomes.

Intervention type (number in study)

The primary goal is to improve health by forestalling the development of illness in the first place.

Programs to promote wellness activities and health screenings or to prevent falls. These interventions do not include programs to prevent adverse events.

Interventions that seek to ensure that clinical staff adhere to specified treatment protocols or other forms of standardized practices. These interventions seek to modify care processes by changing where care is delivered, how care is organized or structured, or who delivers care.  Multi-site intensive care unit telemedicine program.

A team of clinicians use a four-step mobility protocol to regularly assess the functional and clinical status of intensive care unit patients with respiratory failure.

Appendix III: What Makes Some Study Designs More Rigorous Than Others

The methodological literature on assessing the effect of interventions places a major emphasis on study design for identifying those studies that have the capacity to assess an intervention’s effect on an outcome. The key strength of rigorous study designs is that they can take account of other factors that could affect the outcome of interest, and thereby isolate the effect of the intervention itself.

Randomized controlled trials (RCTs) are widely considered to be among the most rigorous types of study designs because their basic structure inherently minimizes the potential impact of confounding factors on their results. RCTs accomplish this by randomly allocating study participants to groups that either receive the intervention—generally referred to as intervention or treatment groups—or do not receive the intervention—the control groups. The consequence of random allocation is that the only systematic difference between study participants in the two groups is exposure to the intervention. Thus, the effect of all other factors is the same on the two groups and therefore neutralized in making comparisons between the intervention and control groups.

A second design type, known as the controlled before and after study, can be used in situations where the random allocation of study participants between intervention and control groups required for an RCT is not feasible. Controlled before and after studies use data collected from separate treatment and control groups, both before and after the intervention’s implementation, to help to separate the effect of the intervention from that of other factors at work over that time period. In this design type, the control group is generally chosen in a way that is likely to produce a group that is broadly similar to the treatment group prior to the implementation of the intervention. However, methodologists generally recommend an explicit analysis to compare the intervention and control groups used in controlled before and after studies in order to demonstrate that they were in fact similar before the intervention took place.

A third design type, an interrupted time series study, is not based on a comparison of intervention and control groups. Instead, it tracks an outcome of interest over time with measurements taken at many different time points both before and after the intervention. The multiple data points from before the implementation of the intervention enable analysts to take account of the impact of other factors on the outcome and thereby isolate the intervention’s effect on that outcome. The interrupted time series design works best when there are data from a substantial number of different time points, both before and after implementation of the intervention.

Other types of study designs cannot isolate the effect of an intervention from that of other factors because they provide no separate information on what would have happened without the intervention. For example, in a simple pre/post study all one has is a measurement of the outcome before implementation of the intervention and a measurement of the outcome after the intervention. The observed difference reflects all the factors (including the intervention) affecting the outcome over that time period. Because confounding factors could potentially affect the outcome in either the same or the opposite direction as the intervention, the actual effect of the intervention itself could be either greater or smaller than the simple pre/post difference. Even the direction of the intervention’s effect, to increase or decrease the outcome, could be the opposite of the overall change from pre to post. That is why the results of a pre/post study generally cannot be relied on to provide even an approximation of what the likely effect of a health care intervention is on quality of care and costs.

Appendix IV: Key Questions for Assessing Evidence from Studies of Interventions That Seek to Enhance Value

The following three tables provide a set of questions that are intended to help policymakers and others find the information needed to assess the strengths and limitations of evidence drawn from studies of health care interventions that seek to enhance value relating to their impact on quality of care and costs. The three tables focus on the three broad dimensions described in the body of this report: (1) the credibility of evidence that attributes changes in quality of care and costs to the intervention, (2) the applicability of study results for broader populations of interest, and (3) the intervention’s capacity for widespread replication.

Each table lists a series of questions that highlight key information for assessing the evidence produced by relevant studies along with guidance on how to look for that information in published reports. Answers to most of these questions may be found in relevant sections of those reports; if not, one can ask the investigators who conducted the studies. While this set of questions is selective and does not cover every potential methodological issue, the information it calls for should provide policymakers a basis for making an informed assessment of the overall credibility and scope of the available evidence regarding the apparent impact of these interventions on quality of care and costs, as well as the demonstrated capacity of those interventions for widespread replication.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the individual named above, Jessica Farb, Assistant Director; Kristin Ekelund; Krister Friday; Katie Mack; and Eric Peterson made key contributions to this report.