Challenges in Systematic Reviews of Economic Analyses

Economic analyses, including cost-effectiveness analyses, costutility analyses, costbenefit analyses, and cost-minimization studies, can provide valuable information for health care decision makers (see Glossary). Economic analyses can be performed within the context of a clinical trial or other study, or they can be based on a model, or a combination of the two. Economic analyses that incorporate models allow investigators to explore multiple questions (Table 1). Most important, models integrate information about effectiveness and costs and thus can provide a measure of health care value. As more economic analyses are produced, researchers and policymakers need to have methods to synthesize and interpret the results of multiple analyses that address a single issue; systematic review offers a framework for doing this. Table 1. Potential Aims of Systematic Reviews of Economic Analyses Systematic reviews of economic analyses pose special challenges for those who perform reviews and those who use them. Traditional techniques of meta-analysis are not appropriate for economic analyses, which are themselves syntheses, and hence should not be combined as one might combine the results of different randomized trials. Instead, systematic reviews of economic modeling studies are most useful for comparing and contrasting how different investigators have chosen to structure their models and estimate key variables. They can also clarify how results differ between studies based on these different assumptions. Identifying sources of variation across studies can help individual decision makers determine which studies best apply to their particular settings and can also guide future research by identifying important areas of uncertainty. Systematic assessment of study quality can help reviewers interpret individual study results (1, 2). We examine key methodologic challenges in performing systematic reviews of economic analyses, based on our experiences and those of other experts in the field (1, 3). To illustrate our general principles, we also describe our experience in performing a systematic review of colorectal cancer screening for the Agency for Healthcare Research and Quality (AHRQ) and the U.S. Preventive Services Task Force (USPSTF) (4, 5). Challenge: Identifying Economic Analyses Literature searches to identify economic analyses should generally begin with the British National Health Service Economic Evaluation Database (NHS EED) (available at www.york.ac.uk/inst/crd/nhsdhp.htm), which systematically identifies and catalogs health-related economic analyses. The NHS EED uses a rigorous set of search terms in 4 main databases: Current ContentsClinical Medicine (1994 onward), MEDLINE (1995 onward), CINAHL (1995 onward), and EMBASE (2002 onward). In addition, they hand-search key journals and incorporate technology assessments and working papers from a range of research institutes to assure coverage. Their search strategies are available at www.york. ac.uk/inst/crd/nfaq2.htm. When using the NHS EED, reviewers should combine topic search terms with the economic evaluations tab to identify relevant articles. Assistance from a research librarian will help assure good search-term coverage. To identify other potentially eligible studies, including older analyses, reviewers can apply the full search terms recommended by the NHS EED to studies published before 1995, or they can use an abbreviated search strategy. We have found that using the Medical Subject Headings terms costs and cost analysis and cost-benefit analysis works well for identifying relevant studies in prevention and screening. Another useful resource is the CEA registry at the Harvard School of Public Health (available at www.hsph.harvard.edu/cearegistry). In addition, one may wish to query experts in the field and hand-search relevant article bibliographies. Challenge: Selecting Economic Analyses for Review Effectively selecting economic analyses for review begins with carefully defining key questions; taking a methodical approach to the research topic facilitates stating them in ways that a systematic review truly can address. For the USPSTF and its partner Evidence-based Practice Centers (EPCs), the main step in this process is development of an analytic framework and related key questions (6). In this process, the USPSTF and the EPC specify the health or intermediate outcomes of interest, the types of comparisons to be made, and the populations to be included. Systematic reviews of economic analyses can inform a variety of clinical or policy questions (Table 1). Once reviewers or decision makers specify the research questions, reviewers should develop inclusion (eligibility) criteria a priori so that they can identify studies most likely to produce unbiased, generalizable, and useful information for the issues at hand. In practical terms, determining how stringent to be in setting eligibility criteria is often difficult until the number of potential studies has been determined. Regardless of the number of studies available, analysts must apply some minimum quality criteria (1, 2). In addition, reviewers may set criteria relating to the analytic perspective taken (for example, societal, payer, health care system, or patient). Outcomes are a significant element of eligibility criteria. Cost-effectiveness analyses and costutility analyses that measure generic outcomes such as cost per life-year or quality-adjusted life-year (QALY) gained, respectively, are generally more valuable than studies that measure cost per event prevented because the former 2 types yield metrics such as QALYs that can be compared across different clinical conditions. Studies examining disease-specific outcomes may be appropriate if decision makers are interested in comparisons relevant to a single intervention or condition. This might include determining the optimal frequency for delivering a particular screening intervention (for example, mammography or cervical cancer screening) or examining the best way to address a certain variable, such as adherence to clinical advice about screening (for example, for colorectal cancer). In economic analyses, usually only a few studies are relevant for any given topic, and the number of high-quality studies is usually lower. Thus, reviewers must set eligibility criteria to exclude low-quality analyses while still retaining enough studies, if possible, to evaluate important differences in model structure and inputs for key variables. This approach of using the best available evidence typically requires an iterative process; in this, analysts may set final selection criteria only after initially assessing of the quantity and quality of the extant literature (7). Challenge: Critically Assessing Economic Analyses Critical appraisal begins by first systematically identifying, cataloging, and analyzing key study features of the included articles (Table 2). Several checklists for appraising study reporting and quality have been developed (1, 2, 8); however, a recent systematic review found that economic analyses published in the past 10 years frequently have important methodologic flaws (9). In the following paragraphs, we describe key areas for assessment and appraisal in greater depth, including model type and structure, analytic perspective, time horizon, and ways to address uncertainty. Table 2. Common Data Elements To Be Identified from Individual Studies Model Type and Structure Those investigating the same clinical question may structure their models differently. For example, one investigator may use Markov analysis, whereas another investigator may use microsimulation or a discrete-events analysis for the same clinical question. Even within a particular model type, differences in how the model is structured and how information on natural history is used can have important effects on results. How much such differences will affect the results of the model, apart from differences in input variables, is not clear. Analytic Perspective Studies may differ in the perspective they use. Those using the societal perspective may produce results different from those done from the payer or the patient perspective because they include different costs and effects. Many experts prefer the societal perspective because it is the broadest and includes the greatest range of costs and effects. Even studies done from the societal perspective, however, commonly fail to address certain costs, such as those associated with patient time requirements or changes in economic productivity. Time Horizon Different models may use different time periods over which benefits, adverse effects, and costs of a given intervention accrue. Differences in time horizon can lead to important differences in outcomes, particularly when the benefits and adverse effects occur at different points in time. For example, carotid endarterectomy for stroke prevention leads to a short-term increase in adverse events from perioperative complications, but it reduces later events (10). Ways To Address Uncertainty Another important shortcoming of published economic analyses is the limited degree to which many economic analyses have examined and represented uncertainty. Uncertainty should be considered at a minimum of 2 levels: uncertainty with respect to the values used for input variables, and uncertainty with respect to the effect of combining multiple estimated variables. Many studies address the first type of uncertainty with sensitivity analysis of a limited number of variables over an often limited range of values; in some cases, the model structure might permit multiway sensitivity analyses. Few studies, however, consider the higher-level uncertainty that comes from the combined effects of the different variables, including best- and worst-case scenarios. For example, few economic analyses report confidence or credible intervals around their main results; omitting them may engender a false sense of p