Army Corps of Engineers: Evaluations of Flood Risk Management Projects Could Benefit from Increased Transparency
Released: 2019-11-26T00:00:00

Letter

Background

Most communities in the nation experience some kind of flooding, which may occur after substantial spring rains, heavy thunderstorms, winter snow thaws, or heavy storms over a large body of water. Flood risk management includes the appropriate use of structures such as levees and floodwalls, as well as nonstructural measures such as land acquisition and structure relocation, to reduce the risk of loss of life, reduce long-term economic damage to the public and private sectors, and improve the natural environment. Flood risk management is one of the Corps’ three primary missions. For fiscal years 2015 through 2017, the Corps requested more than $3 billion for 71 construction projects that fell within its three missions, of which the largest amount—$1.33 billion—was for 33 construction projects in the flood risk management mission.

Corps of Engineers Organization

Located within the Department of Defense, the Corps has both military and civilian responsibilities. Through the Civil Works Program, the Corps plans, constructs, operates, and maintains a wide range of water resources development projects such as navigation and flood risk projects. The Assistant Secretary of the Army for Civil Works, appointed by the President and confirmed by the Senate, sets the strategic direction for the program and has principal responsibility for the overall supervision of functions relating to the Army’s Civil Works Program. The Chief of Engineers, a military officer, is responsible for execution of the civil works and military missions. The Civil Works Program is organized into three tiers: headquarters in Washington, D.C.; eight regional divisions; and 38 local district offices. (See fig. 2.)

Corps Water Resources Development Projects and Nonfederal Sponsors

The Corps develops water resource projects, including flood risk management projects, in conjunction with nonfederal sponsors such as state and local governments. According to Corps guidance, the planning process for these projects begins with the nonfederal sponsor identifying a problem and approaching the Corps to help develop a solution. Upon congressional authorization for a study and appropriations to fund it, the Corps and the nonfederal sponsor establish an agreement to conduct a feasibility study for a potential project. The Corps initiates a feasibility study by forming a project team comprised of Corps engineers, economists, planners, and possibly other specialists such as nonfederal consultants to conduct the study. The planning process the Corps uses to carry out feasibility studies is described later in our report. Nonfederal sponsors are to participate in the planning process, as well as remain involved through project design, construction, and post-project operations and maintenance. For example, for projects in which the Corps constructs infrastructure such as a flood wall, the nonfederal sponsor is to assume responsibility for monitoring and maintenance costs associated with the flood wall after its construction.

Corps Water Resources Development Planning Guidance

The U.S. Water Resources Council’s Principles and Guidelines outlines the principles and procedures the Corps is to follow for planning water resources development projects, including those with flood risk management objectives. The Principles and Guidelines states that the federal objective of water resources development projects is to contribute to national economic development while protecting the nation’s environment. The Corps implements the planning process outlined in the Principles and Guidelines by conducting feasibility studies for proposed water resources development projects. The Corps’ Planning Guidance provides detailed guidance on how to implement the general process outlined in the Principles and Guidelines for planning water resource projects. The Corps’ National Economic Development manuals provide supplemental guidance for the economic analysis of different types of projects—including flood risk management—and how to evaluate the benefits and costs associated with each type of project.

To identify the beneficial and adverse effects of each alternative plan considered for a project, the Corps uses four categories of analysis established in the Principles and Guidelines: (1) National Economic Development, (2) Environmental Quality, (3) Regional Economic Development, and (4) Other Social Effects, as shown in table 1. The Corps’ Planning Guidance states that feasibility studies may evaluate the effects of alternative plans using the four categories of analysis, but the evaluations under two categories—National Economic Development and Environmental Quality—must be presented in each feasibility study. According to the Corps’ Planning Guidance, the National Economic Development category requires an economic analysis of each plan’s potential economic benefits and costs in monetary terms, while the Environmental Quality category evaluates each plan’s potential nonmonetary effects such as effects on habitat quality and quantity. The Planning Guidance states that using these categories of analysis provides a basis for determining which alternative plans should be eliminated from consideration, modified, or selected for further analysis.

The Corps’ Multi-step Planning Process Identified and Evaluated Benefits, Costs, and Effects of Proposed Flood Risk Management Project Alternatives The Corps Identified and Evaluated the Economic, Environmental, and Other Effects of Proposed Alternatives Using a Multi- step Feasibility Study Process

The Corps’ followed the six-step planning process for water resources development projects outlined in its Planning Guidance to identify and evaluate the beneficial and adverse effects of alternative plans for flood risk management projects and select a recommended plan for the eight feasibility studies we reviewed. In the initial three steps of the planning process, the Corps (1) identified the objectives and other parameters of the project; (2) inventoried and forecasted water and related land resources conditions within the planning area; and (3) formulated alternative plans for further consideration. In the final three steps of the planning process, the Corps (1) evaluated and analyzed each alternative plan for its economic, environmental, and other effects, (2) compared the alternative plans to each other, and (3) selected a recommended plan. Corps officials told us that this six-step process is the basic template for planning water resources development projects across all Corps mission areas. (See fig. 3.) For each of the eight studies we reviewed, the Corps followed this template and addressed each of the six steps in planning the proposed flood risk management project, as we describe below.

Each study identified objectives, problems, opportunities, and constraints for the project. According to the Corps’ Planning Guidance, identification of problems and opportunities is the foundation for scoping the planning process and should begin as soon as practicable after the decision to initiate a feasibility study. Planning objectives describe the desired results of the process by solving the problems and taking advantage of the opportunities identified. Constraints are restrictions that limit the planning process and are unique to each study. Such constraints can be, for example, limitations imposed by policy or law. All of the studies we reviewed had the objective of reducing or managing flood risk and damages in response to problems such as historic river or stream flooding in the planning area. The studies identified opportunities, such as improving the community’s understanding of flood risk and resiliency from flood events. The studies also identified constraints, such as the need for the plan to incorporate extensive transportation infrastructure within some of the planning areas.

Step 2: Inventory

The studies inventoried historic and existing water and related land resource conditions and forecasted future conditions within the planning area relevant to the identified problems and opportunities from step one. According to the Corps’ Planning Guidance, the Corps is to use quantitative and qualitative descriptions of critical resources in the planning area to define existing and future without-project conditions— that is, the conditions if no project is constructed. The defined without- project conditions provide the basis from which the Corps formulates alternative plans and assesses impacts. The studies we reviewed inventoried the existing conditions for the planning area. This inventory included geology, groundwater, surface water, hydrology, water quality, biological resources, cultural resources, land use, recreation, air quality, climate change, transportation, public health and safety, public services, utilities, socioeconomics, and environmental justice. The Corps used these existing conditions to forecast the future without-project conditions, such as increasing flood risk for residential and industrial development, culturally significant communities, or specific infrastructure such as a regional wastewater facility.

Step 3: Formulate

The studies formulated alternative plans for the project, including a range of structural and nonstructural measures and strategies. According to the Corps’ Planning Guidance, an alternative plan consists of a system of management measures, that is, structural and/or nonstructural measures, strategies, or programs formulated to meet the project objectives subject to the planning constraints. The Corps is to identify a range of alternative plans at the beginning of the planning process, screen the plans, and refine them in subsequent iterations throughout the planning process. The Planning Guidance also states that as the Corps develops the alternative plans, it must consider the criteria of completeness, efficiency, effectiveness, and acceptability. In the eight studies we reviewed, the Corps followed an iterative approach to identify measures and form alternative plans. For example, the studies generally identified an initial array of structural and nonstructural measures for conceptual screening, followed by the grouping of viable measures into alternative plans for screening under the criteria, resulting in an array of plan alternatives for more detailed analysis of the beneficial and adverse effects (monetary and nonmonetary) of each. According to Corps officials, flood risk management studies must consider a minimum of two plans— no action and an alternative—and one of the plans considered must be nonstructural. All eight studies we reviewed adhered to this requirement and considered a variety of alternative plans for each proposed flood risk management project.

Step 4: Evaluate

The studies evaluated each alternative plan—including its beneficial and adverse effects—through a comparison of the with-project and without- project conditions. According to the Corps’ Planning Guidance, evaluation consists of (1) forecasting the most likely with-project (e.g., with the alternative plan constructed) condition expected under each alternative plan; (2) comparing each with-project condition to the without-project condition and documenting the differences between the two; (3) characterizing the beneficial and adverse effects; and (4) identifying the plans that will be further considered in the planning process. The studies we reviewed used the categories established in Corps guidance— the National Economic Development and Regional Economic Development categories for monetary benefits and costs and the Environmental Quality and Other Social Effects categories for nonmonetary (quantitative and qualitative) effects—to evaluate and display the beneficial and adverse effects of plan alternatives. The categories and specific types of monetary benefits and costs and nonmonetary effects that the Corps evaluated varied for each study depending on the planning area conditions and the measures and strategies included in the alternative plans. In the studies we reviewed, the economic analyses of monetary effects generally resulted in an estimated net dollar value of benefits (benefits minus costs) expected with each alternative in place, while the analysis of nonmonetary effects generally resulted in a Corps judgment about the net qualitative effect or net quantitative effect (e.g., net units of habitat created) for each alternative.

Step 5: Compare

The studies compared the alternative plans based on the economic analysis of benefits and costs and on the evaluations of environmental and other effects. According to the Corps’ Planning Guidance, the alternative plans (including the no-action plan) are to be compared with each other, with emphasis on the outputs and beneficial and adverse effects that will have the most influence in the decision-making process. Such a comparison is to include monetary and nonmonetary benefits and costs and identify and document trade-offs to support the final recommendation. In the studies we reviewed, the Corps compared project effects in a variety of ways, for example, in a series of narratives describing the beneficial and adverse effects of alternative plans, or a grid for side-by-side comparison of selected effects for plan alternatives. In some studies, this comparison included an incremental process in which the Corps considered incorporating additional measures or approaches into an alternative to further optimize the trade-off between beneficial and adverse effects. The result of this step was a final group of plans that the Corps considered for recommendation.

Step 6: Select

The Corps recommended a plan based on the comparison of the alternative plans. According to the Corps’ Planning Guidance, the Corps should recommend a single alternative plan that must be shown to be preferable to taking no action (if no action is not recommended) or implementing any of the other alternatives considered during the planning process. In the studies we reviewed, the recommended plan and the rationale for its selection were identified in the analyses and underwent internal technical review at the district, division, and headquarters levels. The Chief of Engineers signed and submitted the proposed plan for the project—known as the Chief’s Report—to the Office of the Assistant Secretary for review, and the Secretary submitted the report to Congress for authorization.

The Corps Used Economic Analyses in Its Feasibility Studies to Evaluate Project-Specific Benefits and Costs and Used Additional Analyses to Evaluate Other Effects

All eight of the studies we reviewed included step 4 of the Corps’ six-step planning process: an economic analysis of the benefits and costs of each proposed project as well as an Environmental Quality analysis, as called for in the Corps’ Planning Guidance. The inclusion of the other two types of analyses—Regional Economic Development and Other Social Effects—are not required, but six of the studies included them. The Principles and Guidelines provide the Corps with general flexibility to choose which benefit and cost categories to include in these analyses. The Corps’ Planning Guidance states the federal government’s and project’s objectives guide the planning process, which includes benefit and cost category selection.

The monetary benefits most commonly included in the economic analyses of the Corps feasibility studies we reviewed were reduced damages and emergency costs avoided, as shown in table 2. The Corps included reduced damage benefits in each of the eight studies we reviewed. Reduced damages result from actions such as performing physical modifications to property designed to reduce the frequency of flood damage, relocating structures, or installing flood warning and preparedness systems. For example, a feasibility study for a proposed project in the New York District outlined a plan to modify channels that line the Mamaroneck and Sheldrake Rivers with the goal of reducing the risk of life and property damage within the Village of Mamaroneck. The Corps also included emergency costs avoided as benefits in four of the eight studies we reviewed. Emergency costs include expenses resulting from a flood that otherwise would not be incurred. For example, some of the emergency costs avoided for this proposed project in the New York District included the costs of evacuation, reoccupation, flood fighting, and increased operations, police, fire, and military patrol. Depending on the potential effects of the plan alternatives considered, some studies included monetary benefits from recreation, reduced maintenance costs, flood insurance administrative savings, or reduced transportation disruptions in their economic analyses, but these were not commonly considered in the studies we reviewed.

The Corps considered a variety of monetized costs in its economic analyses for feasibility studies we reviewed, as shown in table 3. Among the most commonly included costs in each of the eight studies were for construction; operation, maintenance, repair, replacement, and rehabilitation (OMRR&R); and real estate. Specifically:

Construction costs. These are the direct costs of installing project measures. For example, the Honolulu District study included the costs of constructing six in-stream debris and detention basins above a watershed, floodwalls along a canal, an earthen levee, and two pump stations.

OMRR&R costs. These represent the current monetary value of materials, equipment, services, and facilities needed to operate the project and make repairs, rehabilitations, and replacements necessary to maintain project measures in sound operating condition during the period of analysis. For example, the Wilmington District study included OMRR&R costs for conducting visual inspections of the levee, mowing twice a year, and conducting video inspections of pipes and culverts every 5 years.

Real estate costs. These include activities such as buying out residential structures and demolishing them. For example, the San Francisco District study included real estate costs to acquire approximately 900 acres of city-owned land for ecosystem restoration and levee, road, and temporary work easements.

Depending on the potential effects of the plan alternatives considered, some of the studies we reviewed included environmental costs; relocations; planning, engineering, and design; and the costs for cultural resource preservation, recreation, and flood warning systems.

In addition to the required economic analysis of benefits and costs, the Corps included other analyses to evaluate monetary and nonmonetary project effects in the flood risk management feasibility studies we reviewed. These included the Environmental Quality, Regional Economic Development, and Other Social Effects analyses. All the studies we reviewed included the Environmental Quality analysis; six studies included the Regional Economic Development or Other Social Effects analyses, as shown in table 4. Corps officials said the additional analyses were included in studies because the analyses were needed to determine the best project design, help make planning decisions, or respond to local sponsors’ preferences.

Examples of some additional analyses conducted in different districts include the following:

Regional Economic Development effects. In the Sacramento District study, the Corps considered ways reduced flooding could increase local business revenue and short-term construction employment but reduce employment because of loss of damage to businesses, among other effects. The Corps also considered how its expenditures for various services and products during the project were expected to generate additional economic activity, such as through additional jobs, income, and sales. In this case, the Corps estimated the project might add 18,930 jobs in the region. According to a 2011 Corps handbook, considering Regional Economic Development effects can provide a better understanding of the overall impact to the region. Doing so also examines the potential impacts mainly to the localized or regional economic area, instead of the nation as a whole.

Other Social Effects. In the Wilmington District study, the Corps considered security of life, health, and safety; preservation of historic significance; and the impacts to cultural resources. According to a 2009 Corps handbook, considering the Other Social Effects analysis has great potential value for better ensuring that water resources solutions address a broad array of issues and concerns that better meet stakeholder needs and expectations.

The Corps’ Evaluations Used Economic Analyses to Identify Project Alternatives with Greatest Net Benefits but Relied on Other Analyses for Some Recommendations

In most of the studies we reviewed, the Corps recommended the alternative plan with the greatest net economic benefits based on the results of its economic analyses. In some cases, however, the Corps relied on other analyses to address different project objectives or the preferences of the local nonfederal sponsors. The Corps’ Planning Guidance directs that the project alternative with the greatest net economic benefit, consistent with protecting the nation’s environment, be selected for recommendation unless an exception is granted. The Assistant Secretary of the Army for Civil Works has the authority to grant exceptions if federal, state, local, or international concerns exist. The Planning Guidance states that projects may deviate from the alternative plan with the maximum net benefits if requested by the nonfederal sponsor and approved by the Assistant Secretary of the Army for Civil Works. Such plan alternatives are referred to by the Corps as the locally preferred plan, with the nonfederal sponsor responsible for any project costs in excess of the costs of the plan with the highest net benefits.

The Corps conducted economic analyses in each of the eight studies we reviewed, resulting in a wide range of monetary benefits and costs for the recommended project plan alternatives. Table 5 shows the monetized benefit and cost information that helped the Corps select recommended plans in the eight studies. The annualized project benefits ranged from approximately $500,000 to $210.6 million, and annualized project costs ranged from about $1 million to $65 million, resulting in annual net benefit estimates ranging from approximately -$500,000 to $146 million.

For five of the eight studies we reviewed, the Corps primarily used the results of the economic analysis of benefits and costs to recommend a plan with the greatest net benefits from among the alternatives, in accordance with the Planning Guidance. These five studies were with the New York, Honolulu, Sacramento, Nashville, and Kansas City Corps districts. Three of the eight studies we reviewed relied on other analyses as allowed under the Planning Guidance to address different project objectives or the preferences of the local nonfederal sponsors. Corps officials said they relied on other analyses when needed to determine the best project design, help make decisions, or respond to local nonfederal sponsors’ preferences. Specifically:

Chicago District. The Chicago District recommended a project based on two separate analyses. Specifically, the project team recommended an alternative plan based on an economic analysis for the flood risk management objective and separate analyses for an ecosystem restoration objective. A Corps document stated that by doing so, the proposed project would help both manage flood risks and restore ecosystems in the watershed. In addition, the study said the recommended plan attempts to maximize the net benefits and find balance between both objectives.

Wilmington District. The Wilmington District study indicated that the Corps recommended the locally preferred alternative plan, after receiving approval to do so, instead of the alternative plan with the greatest net benefits at the request of the nonfederal sponsor. The locally preferred alternative plan was recommended so it could incorporate consideration of potential other social effects, such as life and safety risk, and regional economic development, such as employment created during and after construction. By doing so, the study indicated Corps officials responded to local priorities and the recommendations provided by the President’s Council on the Future of Princeville, North Carolina. According to the study, the Corps considered impacts to community cohesion, cultural and historical values, local per capita and household incomes in comparison to national averages, and other factors not captured in an economic analysis.

San Francisco District. The San Francisco District study indicated that the Corps based its alternative plan recommendation on a combination of multiple objectives and local preference. The recommended alternative plan’s objectives included reducing the risk of tidal floods as well as restoring the ecosystem to tidal marsh habitat. The Corps selected the recommended alternative plan because the nonfederal sponsor wanted to provide additional transitional habitat and greater flood risk management for Federal Emergency Management Agency accreditation over the 50-year study period. Specifically, the local preference was to build the levee about 3 feet higher than the plan with the greatest net benefits— thereby potentially reducing public health and safety risks associated with flooding more than the alternative plan with the greatest net benefit.

Selected Corps Economic Analyses Were Generally Consistent with Best Practices, Although Some Practices Were Not Fully Used

The economic analyses for the eight studies we reviewed generally met three of the five key methodological elements and partly met two key elements—analysis of effects and transparency. Our Assessment Methodology for Economic Analysis (Assessment Methodology) identifies five key methodological elements to the baseline structure of an economic analysis. For the analysis of effects element, the Corps has either taken steps to address certain best practices or indicated the agency is limited in adopting other practices due to statutory requirements. For the transparency element, Corps officials acknowledged that transparency could be improved through its review process.

The Economic Analyses in All Eight Studies Generally Met Three Key Methodological Elements

Objective and Scope

According to our Assessment Methodology, an economic analysis should state the action examined and the justification for the action. In addition, the objective of the analysis should be stated; the scope of the analysis should be designed to address the objective; and the analysis period should be long enough to encompass the important economic effects of the proposed action.

We found that all eight analyses generally met this key element. For example, all eight economic analyses indicated that the actions examined included the evaluation of flood risk management improvements for resolving flooding problems. In addition, the analyses provided specific planning objectives, such as to reduce flood risks in the relevant watershed over the 50-year analysis period and to improve the quality of life for local neighborhoods. Furthermore, all eight analyses used a 50- year analysis period to analyze benefits and costs—a period that should be long enough to encompass important economic effects, though several studies assumed that economic conditions would remain the same over that time period. For example, the analysis for the Honolulu District’s flood risk management study assumed that the inventory of homes and businesses in the flood plain would not change over the 50- year analysis period. According to the analysis, the project area includes sites that are underutilized or not fully developed, but uncertainty about how development might proceed made it difficult to project what changes might occur. The study acknowledged that changes in the business and residential makeup of the watershed over the 50-year period would occur but that the exact nature of these changes could not be projected with any degree of certainty.

In addition, two of the eight studies involved multipurpose projects and specified additional economic-related objectives for ecosystem restoration. For example, the analysis for the San Francisco District’s feasibility study indicated that it was designed to evaluate and compare the economic justification and cost effectiveness of various measures to reduce flood risk and provide ecosystem restoration in South San Francisco Bay. Similarly, the Chicago District’s study indicated that in developing an ecosystem restoration plan, undeveloped lands throughout the watershed were evaluated to determine whether cost-effective aquatic ecosystem restoration at that site was possible and what measures would provide the lowest incremental cost per unit of habitat output.

Alternative Identification and Description

Our Assessment Methodology recommends that an analysis used to examine economic effects should identify and compare alternatives. In addition, the analysis should consider a range of relevant alternatives and should justify that the economic conditions specified under each alternative considered represent the best assessment of conditions under that alternative.

We found that all eight economic analyses generally met this key element. For example, all eight economic analyses examined the economic effect of the proposed flood control actions by comparing a range of alternatives, including various structures such as levees or bridge modifications, as well as nonstructural measures such as floodplain management activities or acquisition of land and removal of people from the flood plain. Moreover, the economic analyses in the studies generally described and justified the economic conditions that would be expected under each alternative. For the two studies that also evaluated ecosystem restoration alternatives, the studies considered alternatives for restoring ecosystems.

Documentation

Our Assessment Methodology recommends that the economic analysis be clearly written, include a plain language summary, and provide clearly labeled tables that describe the data used and the results. Also, the analysis should document that it complies with a robust quality assurance process.

We found that all eight economic analyses generally met this key element. For example, all eight economic analyses were generally clearly written and included tables that generally described data and results. In addition, seven of the feasibility studies included a plain language summary. Six of the studies indicated that the analyses complied with a robust quality assurance process, in which the analyses were reviewed at the Corps district and by technical and policy experts in headquarters. Corps guidance indicates that the quality assurance process for feasibility studies involves reviews for technical quality and policy compliance, among other considerations, at the Corps district and in headquarters. Further, three studies indicated that an independent external peer review had been conducted. While one study completed in the Nashville District did not indicate whether the study complied with a quality assurance process, district officials told us a thorough review was conducted that included multiple district quality control reviews, agency technical review and headquarters policy reviews, and an independent external peer review. In addition, a study completed in the Chicago District did not indicate that it had undergone an independent external peer review.

The Economic Analyses in All Eight Studies Partly Met Two Key Methodological Elements for the Analysis of Effects and Transparency Analysis of Effects

Our Assessment Methodology recommends that an economic analysis quantify the important costs and benefits and monetize these quantitative effects using the concept of opportunity cost—the maximum worth of a good or input among possible alternatives. The criterion of net present value, or related outcome measures, should be applied to compare these effects across alternatives. In addition, the analysis should control for inflation and use economically justified discount rates. Where important costs and benefits cannot be quantified, the analysis should show how they affect the comparison of alternatives.

We identified areas in which the studies did not fully align with certain best practices for various reasons, such as the Corps’ concerns about the reliability of available methods and statutory requirements regarding the use of discount rates. These best practices included:

Quantifying and monetizing important benefits and costs. The economic analyses in all eight studies quantified and monetized important benefits and costs associated with each alternative, such as property damage reductions and construction costs. The Corps’ Planning Guidance indicates that studies should consider analyzing loss of life in the Other Social Effects category, in either monetary, quantitative, or qualitative terms. Project alternatives that reduce the risk of flooding or that relocate people from the flood plain may lower the risk that individuals living or working in a flood plain will drown or become injured during flood events. However, the analyses in the eight studies we reviewed generally did not quantify and monetize the effect of project alternatives on loss of life. One of the studies we reviewed quantified these effects, but only for the recommended plan. Specifically, the Sacramento District’s flood risk management study found that the recommended plan, which involved the improvement of an existing levee system, could reduce fatalities during flood events by about 67 percent. Of the other seven studies that we reviewed, six analyses included a qualitative discussion of the effects of alternatives on loss of life, and one analysis did not include an assessment of these effects. A recent National Academy of Sciences study on coastal storm flooding indicated that the practice of quantifying and valuing reductions in loss of life is widespread in the federal government, allowing these risk reductions to be included in the economic analysis. In July 2017, after the eight studies that GAO reviewed were completed, the Corps issued revised guidance requiring flood risk management studies to include a quantitative assessment of loss of life for each alternative when it is a significant factor. Corps officials said they had not attempted to monetize loss of life because of concerns about the reliability of available valuation methods but are monitoring other agencies’ efforts to value these effects and following economic research in the area.

Using net present value criterion. Analyses for seven studies we reviewed compared the flood risk management alternatives and identified the alternative expected to maximize net benefits on a comparable, present-value basis (that is, on an “annualized” basis). However, one economic analysis did not clearly indicate whether the costs associated with the flood risk management alternatives were annualized and therefore comparable to the annualized benefits.

Controlling for inflation and use of economically justified discount rates. Although all the economic analyses in all eight Corps studies we reviewed controlled for inflation by expressing benefits and costs in “real” terms, the discount rates that the studies used to convert future benefits and costs to present values were in nominal terms. In general, real and nominal values are not combined in the same analysis. Specifically, discounting real benefits and costs with a nominal discount rate understates present values when holding all else the same. Corps officials said that they are aware of this inconsistency, but they have no latitude to use a real discount rate because the Water Resources Development Act of 1974 requires the Corps to use nominal discount rates.

Corps officials acknowledged areas in which the eight Corps studies we reviewed partly met the Analysis of Effects key methodological element. However, as noted, the Corps has taken some steps to address one best practice. Specifically, the Corps’ recently revised guidance, which requires quantification of loss of life effects when significant, should allow the Corps to provide decision makers and stakeholders with more precise information about the relative magnitude of these effects in future economic analyses. In terms of the best practice regarding economically justified discount rates, the Corps has not taken steps because it is required to use the statutorily specified nominal discount rates.

Transparency

Our Assessment Methodology recommends that analyses be transparent with respect to their analytical choices, assumptions, and data used. The methodology further recommends (1) evaluating how plausible adjustments to each choice and assumption may impact the estimates of the cost-and-benefit effects and results of the comparison of alternatives and (2) clearly explaining the implications of the key limitations in the data and models used. Where feasible, to ensure transparency, the analysis is to adequately quantify how the statistical variability of the key data elements underlying the estimates of the economic analysis impacts these estimates and the results of the comparison of alternatives.

We found that the studies we reviewed did not fully use some best practices related to transparency. Specifically:

Being transparent with respect to analytical choices, assumptions, and data used. The economic analyses in the eight studies described and justified several, but not all of the analytical choices, assumptions, and data. For example, to approximate the amount of damages to structures at different flood depths, the Wilmington District’s feasibility study relied on standardized “depth- damage curves” developed by the Corps’ New Orleans District. Corps guidance indicates that standardized curves can be used in the absence of regionally developed data. According to the study, data for structures in the study area were unavailable, and flooding characteristics were similar in the two areas, with both study areas covering urbanized and rural areas representing a mix of residential, commercial, and industrial development with similar types of construction. However, other data and assumptions used by the studies in our review were not fully described or justified. For example, in presenting its results for an initial screening of several flood risk management alternatives, the Sacramento District’s economic analysis relied on cost estimates from several different sources, including prior studies and private consultants. The analysis, however, did not explain how the estimates were developed or justify why the estimates were sufficiently reliable for evaluating alternatives.

Clearly explaining the implications of key limitations in the data and models used. With one exception, the economic analyses we reviewed generally did not discuss the implications of key limitations in the models used in the studies. Specifically, the economic analysis for the Sacramento District’s study indicated that the Corps’ Hydrologic Engineering Center-Flood Damage Analysis computer program can overstate damage reduction benefits because of an inability to account for the reduced floodplain occupancy and reduced value of damageable property following a flood event. According to the analysis, by not taking into account the potential for reduced floodplain occupancy, the estimated damage reduction benefits may be overstated, particularly in areas that experience more frequent or severe flooding. To account for this limitation, the Sacramento District’s study reduced the overall value of properties in the floodplain, lowering the average annual benefits for the recommended alternative by about 29 percent. All the other studies used the same program to estimate damage reduction benefits but did not indicate whether this limitation would affect the estimated benefits of the alternatives evaluated in those studies. In accordance with best practices, the Corps’ Planning Guidance indicates that studies should provide adequate supporting documentation to allow reviewers to understand the models and assumptions used to estimate benefits and costs. Corps officials stated that a project team’s analysis may not document every step it took because these are understood among team members, although they may not be apparent to others.

Quantifying the statistical variability underlying the results of the comparison of alternatives. Although the economic analyses for the eight studies analyzed the effects of uncertainty associated with several key inputs in the economic analysis, the studies generally did not report the key estimates (for example, benefits, costs, and net benefits) on a probabilistic basis. For example, the Chicago District’s flood risk management study presented damage reduction benefits for each alternative in terms of its expected values as well as the probability that the benefit estimate would exceed a particular value. However, estimates for costs and net benefits were presented as point estimates, which may imply a greater sense of precision than is warranted. In accordance with best practices, the Corps’ Planning Guidance requires economic analyses to report net benefits and benefit-to-cost ratios both as expected (mean) values and on a probabilistic basis for each alternative; also, for each alternative, the analyses are to present the probability that net benefits are positive and that the benefit-to-cost ratio is at or above one. Corps officials said the analyses generally did not follow this guidance because it may not have been useful in helping to select a project alternative. Nonetheless, Corps guidance states that information about the probability distributions can help decision making by local sponsors, stakeholders, and federal officials by helping to increase their understanding of the uncertainty inherent in each alternative.

In addition, for only one Corps study, the economic analyses included a sensitivity analysis on the discount rate, which is used to convert benefits and costs of the alternatives to present values. Generally, when benefits or costs are separated in time from each other, the difference in timing should be accounted for by discounting benefits and costs. In addition, the specific discount rate may affect the comparison of alternatives. Corps officials told us that they are required to use the statutorily designated discount rate, and their guidance does not require a sensitivity analysis using an alternative discount rate. The officials added that the Office of Management and Budget requires the Corps to compute the benefit-to-cost ratios for recommended plans using a 7 percent discount rate, for budgeting purposes. The results, though, are not reported in the studies, and the 7 percent rate is not applied in the assessment of the net benefits of the alternatives, according to these officials.

Corps officials stated that in general there is a high level of transparency within the project team and with the nonfederal sponsor, but they acknowledged that transparency may not always exist for those outside the team. For example, a project team’s analysis may not document every step it took or assumption it made because these are understood among team members, although they may not be apparent to others. As a result, Corps officials acknowledged that some inconsistency exists in the transparency of the analyses across feasibility studies. Corps officials told us that teams rely on the Corps’ internal process for reviewing all planning products to help ensure the quality of its feasibility studies and analyses. The officials stated that to improve transparency, the Corps could strengthen its internal review process, for example, by adding steps so that all of the important decisions and assumptions made in the analyses are consistently and clearly described. By conducting future economic analyses for potential flood risk management projects so they are more consistent with best practices for transparency, the Corps can better ensure that decision makers and stakeholders are clearly and fully informed about potential economic effects associated with such projects.

Conclusions

The economic analyses included in Corps feasibility studies provide important information about the potential economic effects of flood risk management projects. While the economic analyses the Corps conducted for the eight studies we reviewed were generally consistent with several best practices, the Corps did not fully employ best practices pertaining to transparency. Because the information in the economic analyses can be complex and technical, following best practices for transparency helps ensure that the methods used to develop estimates and conclusions are clearly and fully presented. By conducting future economic analyses for potential flood risk management projects so they are more consistent with transparency best practices, the Corps can better ensure that decision makers and stakeholders are clearly and fully informed about the potential economic effects associated with flood risk management projects.

Recommendation for Executive Action

We are making the following recommendation to the Department of Defense: The Assistant Secretary of the Army for Civil Works should direct the Chief of Engineers and the Commanding General of the U.S. Army Corps of Engineers to strengthen the Corps’ internal review process for feasibility studies by including steps to ensure consistency with best practices for transparency, such as verifying that all of the important assumptions and limitations in models and their implications for the economic analysis are consistently, clearly, and fully described. (Recommendation 1)

Agency Comments

We provided a draft of this report to the Department of Defense for its review and comment. In its written comments, reproduced in appendix I, the Department concurred with our recommendation. The Department further stated that guidance related to ensuring transparency in feasibility studies and reviews already exists, but acknowledged that it can be strengthened and enforced more consistently by specifically identifying transparency as a review criterion. For example, they stated that the Corps plans to establish systematic guidance for meeting the transparency objective in preparing reports, assure transparency through the agency’s quality assurance process, and assess the degree of transparency as part of agency technical review and quality control assessment.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Assistant Secretary of the Army for Civil Works, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and of Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the U.S. Army Corps of Engineers

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Vondalee R. Hunt (Assistant Director), Brad C. Dobbins (Analyst in Charge), Tim Carr, David Dornisch, Juan Garay, Tim Guinane, Gwen Kirby, Keesha Luebke, Jeanette Soares, Sara Sullivan, and Kiki Theodoropoulos made key contributions to this report.