Assessment Methodology for Economic Analysis
Released: 2018-04-10T00:00:00

Letter

Section 1: Economic Analysis

For the purpose of this report, an economic analysis is defined as an analysis that is intended to inform decision-makers and stakeholders about the economic effects of an action. Economic effects (hereafter also called “effects”) commonly include costs, benefits, and/or economic transfers (for example, transfer payments). Action is defined to include a government law, rule, regulation, project, policy, or program. An action may be examined in the context of legislation, regulation, advocacy, agency operations, or in response to certain events (such as a natural disaster, for example). An economic analysis may be prospective, examining an action that could be taken, or retrospective, examining the outcome of an action that has already been taken. Examples of economic analyses include: 1. An economic analysis of the costs of a government program, project, 2. An economic analysis of the benefits and costs of a government rule 3. An economic analysis of the impact of a proposed or existing regulation on regulated entities and consumers. 4. An economic analysis of an action in response to an event (for example, an analysis of a federal response to a natural disaster); 5. A benefit-cost analysis or cost-effectiveness analysis.

Section 2: Key Elements of an Economic Analysis

GAO reviewed handbooks and guidance on economic analysis that have been issued by various government agencies and institutions and consulted with experts. (Appendix 1 details GAO’s objective, scope, and methodology.) In this Section, GAO synthetizes economic elements and concepts embodied in this literature for use by GAO in its assessment methodology for the review of an economic analysis.

GAO identifies five key methodological elements to the baseline structure of an economic analysis. These key elements are: 1. Objective and scope— the objective and scope of the analysis. 2. Methodology— the methodology used to examine the economic effects. 3. Analysis of effects — the analysis of economic effects. 4. Transparency— the transparency of the analysis of economic effects. 5. Documentation— the documentation included in the analysis.

These key elements are standard to the structure of analyses, generally speaking. That is, an analysis is performed to address an objective; the analysis is scoped to address that objective; the analysis adopts a methodology, which is used to analyze the economic effects of interest; and the analysis is transparent and properly documented. The emphasis on transparency is consistent with the final implementation guidelines of the Office of Management and Budget (OMB). “The primary benefit of public transparency is not necessarily that errors in analytic results will be detected, although error correction is clearly valuable. The more important benefit of transparency is that the public will be able to assess how much of an agency’s analytic result hinges on the specific analytic choices made by the agency.”

Having identified key elements to the structure of an economic analysis we synthetized for each key element, economic concepts embodied in the literature that we reviewed. For example, what might we be assessing under the key element: Objective and Scope? We considered economic concepts commonly identified across the documents we reviewed, and incorporated feedback from the experts and agencies with whom we consulted. The documents that we reviewed included, among others, Circulars A-94 and A-4 released by OMB, handbooks for economic analysis from federal and international agencies, such as the U.S. Environmental Protection Agency, the U.S. Department of Defense, the U.S. Department of Transportation, the Organization for Economic Co- operation and Development (OECD), and the United Kingdom’s HM Treasury.

These documents generally outline a methodical structure to an economic analysis. This methodical structure takes the form of a set of issues or sequence of steps to address while conducting the analysis. These issues or steps, in turn, embody economic concepts. For the purpose of developing our assessment methodology, we synthetized and categorized these economic concepts with the key elements that we identified—as listed below. The concepts for each listed key element are not intended to be exhaustive and do not supplant or alter existing requirements for economic analysis. Depending on the context in which an action is examined, GAO’s assessment of a key element could exclude some concepts, or extend beyond the concepts listed for that element. In such cases, GAO’s written assessment of the relevant key element will specify the concepts that were actually considered in the review process.

The five key elements and economic concepts in GAO’s assessment methodology for an economic analysis are: 1. Key Element: Objective and Scope—the objective and scope of the analysis.

The economic analysis explains the action examined and includes a rationale and justification for the action. The analysis states its objective. The scope of the analysis is designed to address this objective. Unless otherwise justified, the analysis focuses on economic effects that accrue to citizens and residents of the United States, and its time horizon is long enough to encompass the important economic effects of the action. 2. Key Element: Methodology—the methodology used to examine the economic effects.

The economic analysis examines the effects of the action by comparing alternatives, using one of them as the baseline. Unless otherwise justified, it considers alternatives that represent all relevant alternatives, including that of no action. The analysis defines an appropriate baseline. The analysis justifies that the world specified under each alternative considered (including the baseline) represents the best assessment of what the world would be like under that alternative. The analysis identifies the important economic effects for each alternative considered, their timing, and whether they are direct or ancillary effects. 3. Key Element: Analysis of Effects—the analysis of economic effects.

Where feasible, the economic analysis quantifies the important economic effects and monetizes them using the concept of opportunity cost. The analysis applies the criterion of net present value, or related outcome measures, to compare these effects across alternatives. It controls for inflation and uses economically justified discount rates. Where important economic effects cannot be quantified, the analysis explains how they affect the comparison of alternatives. Where the equity and distributional impacts are important, the full range of these impacts is separately detailed and quantified, where feasible. 4. Key Element: Transparency—the transparency of the analysis of economic effects.

The economic analysis describes and justifies the analytical choices, assumptions, and data used. The analysis assesses how plausible adjustments to each important analytical choice and assumption affect the estimates of the economic effects and the results of the comparison of alternatives. The analysis explains the implications of the key limitations in the data used. Where feasible, the analysis adequately quantifies how the statistical variability of the key data elements underlying the estimates of the economic analysis impacts these estimates, and the results of the comparison of alternatives. 5. Key Element: Documentation—the documentation included in the analysis.

The economic analysis is clearly written, with a plain language summary, clearly labeled tables that describe the data used and results, and a conclusion that is consistent with these results. The analysis cites all sources used and documents that it is based on the best available economic information. The analysis documents that it complies with a robust quality assurance process and, where applicable, the Information Quality Act. The analysis discloses the use and contributions of contractors and outside consultants.

In summary, GAO identifies five key elements, with associated economic concepts, to the structure of an economic analysis. GAO’s assessment methodology then examines the extent to which an economic analysis properly dealt with these key elements.

Section 3: Assessment Methodology

GAO’s assessment methodology has two steps: (1) an assessment of each individual key element and (2) an overall assessment based on the assessment of the individual key elements. Below, these two types of assessment are discussed. While GAO’s assessment methodology typically considers all five key elements, there may be cases, for example depending on the scope of an engagement, where it may consider only certain key elements. In those cases, GAO may not be able to make an overall assessment.

Step 1: Assessing Each Individual Element

The first step in the review process is an assessment of the extent to which the economic analysis has considered and properly dealt with each key element. For each element, the outcome of the review is a written assessment and an assessment score. The written assessment details the extent to which the analysis considered and properly dealt with the element. To the extent that important limitations are identified in the review, the written assessment describes these limitations. This written assessment informs the assessment score, which is one of three mutually exclusive scores: 1. fully met—that is, the economic analysis has considered and properly dealt with the element; 2. partly met—that is, the economic analysis has only partly considered and properly dealt with the element; 3. not met—that is, the economic analysis has not considered or not properly dealt with the element.

If the outcome is “partly met” or “not met,” the written assessment should describe the limitations of the analysis.

Assessments are made from expertise in economics and with professional judgment. The guiding principles of a review are objectivity, integrity, and compliance with generally accepted government auditing standards (GAGAS). An assessment is contextual—that is, it is conditional on the evidence underlying the action examined, and the context in which it takes place. The assessment is also conditional on the reasonably obtainable information available at the time of the economic analysis that is being reviewed. A caveat may be added to the assessment if new information that would affect it, has become available and is reasonably obtainable since the analysis was made. Should the review of a key element exclude certain concepts, or extend beyond the concepts listed for that key element, GAO’s written assessment of the key element will specify the concepts that were actually considered in the assessment process.

Step 2: Overall Assessment

Once each element has been individually reviewed, an overall assessment is made of the extent to which the economic analysis accordingly informs decision-makers and stakeholders about the economic effects of the action examined. Four outcomes are possible: 1. The analysis informs decision-makers and stakeholders about the economic effects of the action examined. 2. The analysis informs, with caveats, decision-makers and stakeholders about the economic effects of the action examined. (If this is the outcome, the review should describe the caveats in writing.) 3. The analysis needs additional work to inform decision-makers and stakeholders about the economic effects of the action examined. (If this is the outcome, the review should describe in writing the important limitations of the economic analysis.) 4. The analysis does not inform decision-makers and stakeholders about the economic effects of the action examined. (If this is the outcome, the review should describe in writing the deficiencies of the analysis.)

An economic analysis that has fully met all the key elements should be identified as informing decision-makers and stakeholders about the economic effects of the action examined (this is outcome 1). This determination is neither an endorsement of the specific findings and conclusions of the analysis, nor is it a determination that these are correct. For example, a prospective analysis is predictive of a potential result, but cannot definitively determine the result. It is a determination that the analysis is adequately and properly designed, and accordingly, it can inform the public discourse about the economic effects of the action examined.

The written statements added to outcomes 2–4 should refer to the review of the individual elements. The difference between outcomes 2 and 3 is a matter of degree and professional judgment. Generally speaking, the caveats under outcome 2 will be relatively minor or few, whereas the important limitations under outcome 3 are likely to be more consequential. Should the economic analysis suffer from major deficiencies in meeting the key elements, then outcome 4 may be appropriate.

Appendix I: Objective, Scope, and Methodology

Our objective was to identify, for the purpose of developing GAO’s assessment methodology for the review of economic analysis, key methodological elements to the structure of an economic analysis that is intended to inform decision-makers and stakeholders about the economic effects of a public action.

To address this objective, we reviewed existing Circulars issued by the Office of Management and Budget (OMB), handbooks for economic analysis issued by federal agencies, international government agencies and institutions, and established textbooks on economic theory and benefit-cost analysis. We also solicited feedback from economics experts (in academia and public policy) and international audit agencies.

We identified in our document review, five key elements to the structure of an economic analysis. These key elements are: (1) objective and scope; (2) methodology; (3) analysis of effects; (4) transparency; (5) documentation. These key elements are standard to the structure of analyses, generally speaking. That is, an analysis is performed to address an objective; the analysis is scoped to address that objective; the analysis adopts a methodology, which is used to analyze the economic effects of interest; and the analysis is transparent and properly documented.

Having identified key elements to the structure of an economic analysis, we synthetized for each key element economic concepts embodied in the literature we reviewed. For example, what might we be assessing under the key element objective and scope? To do so, we looked for economic concepts commonly identified across the documents we reviewed and incorporated feedback from the experts and agencies that we consulted with. We then categorized these economic concepts across the key elements that we identified.

Among the documents we reviewed in our process were the following:

OMB Circular A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Revised (Oct. 29, 1992). Circular A-94 provides a checklist of whether an agency has considered and properly dealt with all the elements for sound benefit-cost and cost- effectiveness analyses.

OMB Circular A-4, Regulatory Analysis, (Sept. 17, 2003). Circular A-4, released in collaboration with the Council of Economic Advisors, identifies key elements to the structure of economic analyses in regulatory proceedings.

Office of Management and Budget, Office of Information and Regulatory Affairs, Regulatory Impact Analysis: A Primer (Washington, D.C.: The White House). The purpose of the primer is to offer a summary of OMB Circular A-4.

Agency-issued handbooks for economic analysis, such as, for example, those issued by the U.S. Environmental Protection Agency, the U.S. Department of Defense, the U.S. Department of Transportation, and the Organization for Economic Co-operation and Development (OECD). We also reviewed The Green Book: appraisal and evaluation in central government, issued by HM Treasury, Government of the United Kingdom, (London; July, 2011). HM Treasury describes The Green Book as a best practice guide for all central departments and executive agencies, and covers projects of all types and size. The guide applies to appraisals—defined as any analysis used to support a government decision to adopt a new policy, or to initiate, renew, expand or re-orientate programs or projects that would result in measurable benefits and/or costs to the public—and evaluations—defined as retrospective analysis of a policy, program or project at its completion, conclusion or revision.

The National Academies of Sciences, Engineering, and Medicine, Guidelines for the review of Reports of the National Academies of Sciences, Engineering, and Medicine. While these guidelines are specific to the review of reports issued by the National Academies and outline review criteria that apply across a broad range of disciplines, not just economics, they provide review criteria for scientific analysis.

We conducted our work from June 2017 to April 2018 in accordance with all sections of GAO’s Quality Assurance Framework that are relevant to our objectives. The framework requires that we plan and perform the engagement to obtain sufficient and appropriate evidence to meet our stated objectives and to discuss any limitations in our work. We believe that the information and data obtained, and the analysis conducted, provide a reasonable basis for any findings and conclusions in this product.

Third-party Comments

We provided a draft of this product for third-party outside review to experts at various U.S. and international government agencies and specialists. Reviewers provided technical comments which we incorporated as appropriate.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Carol Bray, Timothy Carr, Tim Guinane, Kathleen Scholl, Paola Tena and Elaine Vaurio made key contributions to this report.