More Conformity or More Data

Sherry Glied and her colleagues have done an excellent job of laying out some of the modeling approaches and techniques, underlying assumptions, and results of various studies of health insurance coverage. Their observations about the lack of comparability of results are especially helpful. Their proposed solution of a reference case may be premature, however, and, if widely adopted, stultifying. The truth is that health care research even beyond assumptions remains formative. That is, the results are scattered, not so much because of the investigators’ caprice, but because the models, and particularly the data, do not yet lend themselves to clear and consistent conclusions. This does not mean that the typology of assumptions, for example, is not useful; quite the contrary. But it does mean that when regression results can explain only 10 to 20 percent of the variance, it may be counterproductive to attempt to impose a reference case. Short of specifying a baseline, however, Glied and her colleagues are surely right that more ought to be done to clarify all relevant assumptions and at least to publish enough data to create comparable results, if not to report the results in the several relevant forms. As a case in point, we at the Congressional Budget Office try to consistently report relevant assumptions and underlying data for our many estimates of legislative impact. We recently published a lengthy report on the many twists and turns needed to produce an estimate of the cost of providing pharmaceuticals to Medicare beneficiaries. Indeed, all of our work is in the public domain, and we often use outside reviewers (contrary to the authors’ assertion) and several panels of outside experts to evaluate our simulations. We also go to great lengths to highlight the uncertainty in all our work, from stochastic analyses to graphic portrayals of confidence bands. The irony is not lost on us that the larger the apparent errors are, the more sensitive to uncertainty our audience becomes. Nonetheless, the budget process as currently practiced demands point estimates, which often have large confidence intervals. Although the authors addressed the larger research enterprise, the Congressional Budget Office can do more. We have instituted a formal guest scholar program to help us both test and document critical assumptions. We have a series of monographs on our economic and revenue-forecasting models and invite commentary. We host seminars with interested parties on specific estimating issues like the number of uninsured persons in the United States. We participate in professional meetings and publish journal articles. We engage among ourselves in a formal process of evaluating techniques, data, and assumptions. We meet with not only congressional staff but also nongovernmental persons with a legitimate interest in our work. And we both coordinate and compete with executive branch and other governmental analysts. Despite all this, we need to be mindful of the many inherent limitations of modeling. In particular, estimates of budgetary effects or the projected impact of proposed changes should inform—but should not determine—the value of a policy proposal. When Glied and her colleagues try to increase the reliability and informational value of simulation results, they should remember that simulations sharpen intuitions… but do not make decisions. It would be naive not to recognize that some of our critics have a preferred policy prescription and a preferred outcome. They want more transparency to facilitate their criticism of our work, though not necessarily to better understand the underlying issue. These critics tend to ignore the preponderance of existing evidence (as inconclusive and uncertain as it might be) and the often voluminous documentation we now produce. Often their interest in learning more about our models is driven by a desire to tailor their legislation to minimize an estimate, not to argue for a different assumption. It is important to recognize that the statement by Glied and her colleagues—“most of the parameters are simply not known and could not be known”—is not universally true. As the authors dutifully note, many of the divergent results are due to data that, I would suggest, can best be addressed by developing better data—and a better understanding of the existing data—rather than creating a more uniform way of “how to treat unobservable characteristics.” The case of uninsured persons is a compelling example of the lack of understanding of existing data, even by knowledgeable analysts. That is, the often-cited census results regarding the number of uninsured persons (currently around 40 million) are more misused than used. The survey question itself is open to interpretation and has produced results inconsistent with other surveys and administrative data; for example, millions more people are receiving Medicaid benefits than the survey reports. Perhaps more important, the same census survey that found 40 million uninsured persons also reported that the average duration of their “uncoverage” was around seven months, a discovery that would likely lead to different, or additional, policy prescriptions than are currently being considered. A fair amount of data currently collected by all levels of government are not being used, largely because they are not reported at all or are reported in a way in which they cannot be used. Again, in an example from our own work, we have organized the literally billions of Medicare records of patients for the last decade so that we now have monthly expenditure and diagnostic data for a sample of 5 percent of Medicare beneficiaries. Similarly, a great number of, for lack of a better term, natural experiments are being carried out in the seemingly constantly changing benefits and reimbursement rates of both public and private programs. The implementation of welfare reform, for example, had apparently large effects on Medicaid enrollment. Although data on private programs are admittedly more difficult to obtain and normalize, it is possible to examine the experience of, say, managed care providers who have left the field. Another useful suggestion by Glied and her colleagues is to further segment and examine particular populations. Certainly, various subgroups exhibit different incentives and different behaviors that elude apparent explanation. Beyond the most obvious distinction between public and private programs, each state has a somewhat different Medicaid program, and to date, most studies have not explored these differences. For example, the elderly in this country are insured differently than are the nonelderly, and several of their health characteristics and requirements are different as well. Furthermore, we have ample evidence of regional variations in the use of procedures and the cost of treatments for all ages, variations that are not readily explainable. The delivery of health care in rural settings presents unique challenges, and we should not expect rural and urban patients and providers to act the same even in the face of similar incentives. Glied and her colleagues have performed a great service in highlighting some of the current deficiencies in the analysis of health insurance and, by extension, health care in this country. Moreover, their call for more clarity and transparency of techniques and results is sensible. I believe, however, that it would be more productive to concentrate on other issues, such as the development of more and better data, than to establish a base case while the research remains in such a formative stage.