6. Response to Comments

We appreciate the interest and criticisms of our commentators. Our effort to encourage modelers to engage with one another and with the consumers of modeling results appears to have had an immediate benefit, and we hope this is the beginning of a continuing dialogue. We recognize, as Crippen, Fishman, and Nexon noted, that the Congressional Budget Office (CBO) operates under different constraints than do private modelers. While we do not specifically address how forthcoming the CBO should be about its model, we believe that greater transparency in the private modeling community regarding assumptions would help the CBO refine its model and respond to legislative requests. It would also, as Fishman and Nexon suggested, help the legislative staffers who work with the CBO. Pauly points out a problem, the choice of an appropriate price variable, that we did not include in our sample of issues. Undoubtedly others—as well as new issues—will surface as policies evolve. As Fishman observed, there is ample room for improving models. If modelers begin to make explicit comparisons between their results and assumptions and those of other modelers, making these improvements may not require an outside referee. Developing and maintaining a reference case does not mean imposing conformity on the modelers. We disagree with Sheils and Crippen that using the reference case would cement in place dangerous assumptions. In fact, we rarely recommended for our reference case that modelers use specific values for key parameters. Rather, our guidelines generally suggest that modelers conduct sensitivity analyses that include a “best guess” assumption or report results in a form that can be readily compared with those in the existing literature. We certainly agree with our critics that the modeling process should remain open to new data and research findings. Pauly raised the more general issue that modeling results carry too much weight in the policy process. We agree that the debate over cost estimates should not derail efforts to expand coverage. But budget numbers do matter, especially in health care, in which even appetizer-size programs are enormously costly. Pauly and Herring's (2001) estimate of the gross cost of a moderate fixed-dollar tax credit proposal, for example, is $79 billion per year, a figure larger than the entire annual budget of either the U.S. Department of Agriculture or the Department of Transportation. Faced with costs of this magnitude and working under intense time pressure, Congress demands, as Sheils put it, “one-armed economists,” who provide, if not certainty, than at least consensus, about numbers. Unfortunately, the complexities of modeling and policy are such that uncertainty and error are unavoidable. More data are always helpful, as Crippen pointed out, but even an enormous data collection effort is unlikely to resolve all controversies. Even more data probably would not reveal key parameters, such as the cost of coverage that uninsured people who elected not to buy insurance face, or parameters that have no analogues, such as how firms would behave if a tax credit were available only for nongroup insurance. Thus, even the best trained estimators will, and should, disagree on the numbers, and even the best estimates will have large confidence intervals. The outcomes of all sorts of decisions—from an entrepreneur's decision to open a new coffee shop to the nation's decision to go to war in the Persian Gulf—are uncertain when they are made. The budget process tends to obscure this inherent uncertainty because it demands that a single figure be put on paper, but legislators should recognize that this uncertainty exists when they evaluate a proposal. Indeed, the differences in various models’ estimates can help legislators take this uncertainty into account by illuminating its sources, nature, and extent. Variations across models also may be an effective supplement to the ranges and confidence intervals that the CBO and other estimators already provide. We hope the guidelines we proposed in our article will make both the information and the uncertainty more transparent so that models can enable rather than retard policy innovation.