The Quality of Reporting in Published Cost-Utility Analyses, 19761997

The rapid increase in the number of cost-effectiveness analyses published in the medical literature in recent years has been well documented (1, 2). Moreover, recommendations for conducting, reporting, and using such analyses have been widely promoted to the medical community to help inform decisions about resource allocation (3-7). At the same time, suspicions have persisted about the credibility and comparability of these studies (8-12). Valid comparisons of cost-effectiveness ratios across studies require that the numerators and denominators of the ratios be reported in similar terms and be obtained by using comparable methods (13). Although standards for conducting and reporting cost-effectiveness analyses have been proposed as a way of improving consistency and clarity (4, 14), adherence to recommended protocols has generally been lacking (8, 11, 15). To understand the quality of reporting in the field, we audited published cost-effectiveness analyses in which health effects were measured in quality-adjusted life-years (QALYs) gained. We follow Drummond and colleagues (14) by referring to these studies as costutility analyses. This study goes beyond previous investigations of this kind both in its comprehensiveness and its focus on costutility analysis, which has emerged as a recommended approach for economic evaluation in health care [4]. Our objectives were 1) to examine variations in reporting practices in published studies, 2) to determine whether the quality of reporting has improved over time, and 3) to examine differences in reporting practices by the type of journal and experience of the journal in publishing costutility analyses. Methods Cost-Utility Analysis Cost-utility analysis is a type of cost-effectiveness analysis in which health effects are measured in terms of QALYs gained. In contrast, health effects in cost-effectiveness analysis can be measured in terms of other outcomes, such as cases of disease prevented or years of life gained. Although it is not without critics (16-18), costutility analysis is recommended for several reasons: It captures gains from both prolongation and quality of life in a single measure, it incorporates the value that people place on different health outcomes or their preferences for particulate health states, and it provides a convenient means of comparing analyses of diverse interventions and conditions (Glossary). Literature Search We conducted a systematic search for original costutility analyses through 1997 by using the MEDLINE, HealthSTAR, CancerLit, Current Contents (all editions), and EconLit databases. The databases were searched by using the Medical Subject Headings or text keywords quality-adjusted, QALY, and costutility analysis. These databases and keywords were selected on the basis of their usefulness in our previous work in the field (11) and on discussions with experts in the field. In addition, we screened the titles of more than 6500 articles in two extensive bibliographies of the field (1, 2) for the words cost-effectiveness analysis, costutility analysis, and QALY. Our initial search identified more than 1500 candidate studies. Abstracts of these articles were screened by study investigators to determine whether the paper contained an original costutility estimate. We excluded review, editorial, or methodologic articles; cost-effectiveness analyses that measured health effects in units other than QALYs; and articles in languages other than English. A final set of 228 articles were judged to contain original costutility estimates and became part of our database. A list of the studies and our reporting form is available on our Web site (www.hsph.harvard.edu/organizations/hcra/cuadatabase/intro.html). Developing the Data Auditing Form A standard data auditing form was used to determine the clarity, completeness, and quality of each article. The form was developed on the basis of information from diverse sources, including the checklist for reporting reference-case costutility analyses recommended by the Panel on Cost-Effectiveness in Health and Medicine (5), other published guidelines and recommendations (6, 7, 19-22), and discussions with experts in the field. Pilot testing was conducted on four earlier versions of the form. In each pilot test, two readers analyzed the same four to six articles by using a draft form and then convened to review discrepancies in their findings and to improve the clarity of the form. A detailed set of instructions for readers was developed for the final form to ensure that readers interpreted items uniformly. With regard to reporting practices, we collected data on disclosure of funding sources; reporting of the framing of analyses; reporting of costs; reporting of preference weights; reporting of results; and reporting of key elements in the discussion section. Funding Disclosure We examined whether the authors clearly stated the source of sponsorship and whether funding was from a government agency; nonprofit foundation; pharmaceutical or medical device company; or health care organization, such as a hospital or managed care plan. Framing Framing refers to the adequacy of statements relating to the basic construction of the analyses. We examined whether studies clearly presented the comparator intervention, the study perspective, modeling assumptions, a diagram of the model, and the discount rate used to adjust for future costs and benefits. Cost Estimation We examined whether the study estimated and reported net costs (costs minus potential savings associated with an intervention), whether the original source of valuation for each cost item used in the analysis (for example, whether the cost of a hospitalization was based on published estimates or on billing data from a particular institution) was clearly reported, and whether the authors clearly stated the year and currencies in which monetary units were valued. Estimation of Preference Weights We collected data on whether the authors reported the technique used to measure preference weights (for example, direct elicitation with the time-tradeoff, standard gamble, or rating scale method or use of a generic preference-weighted health state classification system [Glossary]). We also examined whether the source of preference weights was noted (for example, whether patients, clinicians, members of the general population, or the authors' own judgment was used). Reporting of Results We recorded whether the authors clearly stated that they conducted incremental costutility analysis and reported data that could confirm this. We also recorded whether sensitivity analyses were reported for costs, preference weights, estimates of effectiveness, and discount rate. Discussion Section We examined whether the authors discussed study limitations, whether results were compared with costutility ratios from related analyses, and whether the authors discussed the ethical or distributional implications of the analyses. Readers also recorded their subjective assessment of the quality of each article on an interval scale from 1 (low) to 7 (high). This assessment reflects a judgment about the overall quality of the methods, assumptions, and reporting practices used in the analysis. Abstracting the Data Four trained persons, each of whom had graduate education in decision analysis and cost-effectiveness analysis, served as readers for the project. For the auditing process, two of the readers systematically scrutinized each article by using the data auditing form. Each reader participated in pilot testing of the abstracting form. Two readers independently read and audited each article and then convened for a consensus audit to resolve discrepancies. On occasion, a third reader was called on to help settle disputed items. The importance of the consensus discussions and the degree of agreement in the initial scoring of the articles can be seen from examination of the reproducibility of the categorical items on the data auditing form. Kappa statistics were calculated to measure the interrater reliability of each item before consensus was reached. The overall mean was 0.51 (95% CI, 0.45 to 0.57), a rate considered to represent moderate strength of agreement (23). However, the strength of agreement differed across categories of reporting practices; it was highest for results (0.61 [CI, 0.48 to 0.75]) and lowest for discussion (0.40 [CI, 0.28 to 0.52]). We examined trends across three time periods: 1976 to 1989, 1990 to 1994, and 1995 to 1997. These periods were chosen because they represented natural cut-points in the data over time (Figure). We also tested for differences in reporting practices across three types of journals: general medical and surgical, medical subspecialty, and methodologic and policy. Smaller studies have shown that cost-effectiveness analyses published in general clinical journals are more likely to use appropriate analytic methods than are those in specialty journals (8, 24), and we expected to find this tendency in our data. Finally, we tested for differences in reporting practices between high-volume and low-volume journals. On the basis of the distribution of the data (Table 1), high-volume was defined as publication of three or more costutility analyses during the time period under investigation. We hypothesized that high-volume journals would tend to have better reporting practices given their editors' familiarity with such studies. We also performed multiple regression analysis to predict our subjective quality score as a function of journal volume (included as a continuous variable) and journal type. Figure. Increase in the number of published costutility analyses, 19761997. Table 1. Characteristics of 228 Articles Results The number of published costutility analyses has increased markedly over time (Figure). Table 1 shows characteristics of the 228 articles. The studies, most of which were based in the United States, were published in numerous journals and