Army Corps of Engineers: Peer Review Process for Civil Works Project Studies Can Be Improved
Released: 2012-04-09T00:00:00

Letter

Background

Corps Process for Civil Works Project Studies

The purpose of the Corps’ civil works project study process is to inform federal decision makers whether a water resources project warrants further federal investment. The study process is conducted in two phases: reconnaissance and feasibility. In the reconnaissance phase, the Corps conducts an initial evaluation of potential solutions to a water resources problem. If the Corps determines that a project potentially warrants federal investment, it proceeds to a more detailed feasibility study. The feasibility phase generally begins with the signing of a feasibility cost- share agreement between the Corps and the local project sponsor.

Feasibility studies are generally prepared by the Corps’ 38 district offices, with review and oversight provided by the cognizant Corps division office and by headquarters. During the feasibility phase, the Corps formulates and evaluates alternative plans for achieving the project’s objectives and reviews the proposed project to assess whether the benefits of constructing it outweigh its costs. At the beginning of this phase, a feasibility scoping meeting is held to bring the Corps, the local sponsor, and other agencies together to reach agreement on the problems and solutions to be investigated during the feasibility study and the scope of the analysis required. The next step includes an alternative formulation briefing to identify and resolve any legal or policy concerns and to obtain headquarters approval of the tentatively selected plan and to release the draft report to the public. Finally, the draft feasibility report—which presents the study results and findings, including those developed in the reconnaissance phase—is released to the public. At the conclusion of the feasibility phase, the Corps selects a recommended plan for proceeding with the project.

The feasibility report also includes analysis and documentation to meet the requirements of the National Environmental Policy Act (NEPA). Under NEPA, federal agencies are to assess the effects of major federal actions, such as Corps construction projects, that significantly affect the environment and prepare a detailed statement on the environmental impacts of those actions. NEPA has two principal purposes: (1) to ensure that an agency carefully considers detailed information concerning significant environmental impacts and (2) to ensure that this information will be made available to the public. NEPA requires an agency to prepare a detailed statement on the environmental impacts of any “major federal action” significantly affecting the environment. NEPA implementing regulations generally require an agency to prepare either an environmental assessment or an environmental impact statement (EIS).procedures—such as providing the public with an opportunity to comment on the draft EIS for at least 45 days.

NEPA implementing regulations also specify requirements and Corps project studies have historically been subject to various levels of internal and external review under a number of authorities as well as the Chief of Engineers’ responsibility to ensure the quality of Corps studies. For example, in 1902, Congress created the Board of Engineers for Rivers and Harbors, which was the result of efforts to address inconsistent treatment of proposed Corps projects. The board was made up of Corps staff. Until 1992, when Congress terminated the board, it reviewed thousands of Corps studies for civil works projects and made unfavorable recommendations on more than half. At the time the board was abolished, there was concern that too much duplicative review was occurring between the board and other internal Corps review processes. Subsequently, in the Flood Control Act of 1944, Congress established a mechanism for external review of Corps projects by giving the head of the Department of the Interior and the governors of affected states an opportunity to comment on proposed Corps projects before authorization. Furthermore, starting in 1970, under NEPA, environmental impact statements for Corps projects were required to be sent to the heads of other federal agencies and governors of affected states for comment. As a result of the Flood Control Act of 1970, Congress created the position of Assistant Secretary of the Army for Civil Works, who coordinates the review of Corps studies with the Office of Management and Budget (OMB) before they are submitted to Congress.

Increasing Interest in Peer Review of Corps Project Studies

Recent congressional interest in establishing an independent external peer review process for Corps project studies began in the late 1990s, following a series of damaging reports and events, including allegations that the Corps had manipulated information to justify projects. Investigations conducted by NAS and the Army’s Inspector General identified various problems with the Corps internal review process, including the manipulation of economic analysis and potential institutional bias toward large construction projects. Around this time, WRDA 2000 required the Corps to contract with NAS to study and make recommendations concerning the use of peer review for feasibility reports, including recommending potential criteria to determine how to apply peer review. In 2002, NAS released its study concluding that the Corps’ more complex water resources project planning studies should be subject to external, independent review. The study also found that not all Corps project studies necessarily require such review, recommending instead that external peer review be reserved for studies that are expensive, will affect a large area, are highly controversial, or involve high levels of risk. The study estimated that about five Corps projects per year would likely be subject to this level of review. According to the NAS study, criteria for selecting the appropriate level of review should balance the risks and consequences of inadequate review against the resources required for more complex and stringent levels of review. In addition, the study identified several criteria that should be considered in determining the appropriate level of review for Corps studies, primarily that as project magnitude and risks increase, an increasing degree of independence and scope of review are warranted (see fig. 1).

In addition to recommendations related to the appropriate level of review for project studies, the 2002 NAS study made several other recommendations about the Corps’ peer review process. It recommended that peer review results be presented to the Chief of Engineers before a final decision on a project study is made, that the Chief of Engineers respond in writing to each key point of the peer review report, and that peer review be initiated early enough in the Corps’ study process so that review results can be meaningfully incorporated into project design.

After NAS published its 2002 study, OMB in December 2004 issued its Final Information Quality Bulletin for Peer Review citing the Information as well as its general authorities to oversee the quality of Quality Act,agency information, analyses, and regulatory actions. This OMB bulletin established governmentwide guidance on enhancing peer review practices and covers what information is subject to peer review, the selection of appropriate peer reviewers, opportunities for public participation, and related issues. The Corps’ Engineering Circular 1105- 2-408 (EC 408) was issued in May 2005 and established procedures for ensuring the credibility and quality of Corps documents by supplementing its previous review process, including to add external peer review to its review process in special cases where risk and magnitude warrant this level of review.

The Corps faced further criticism after the failure of Corps levees and floodwalls in New Orleans in the wake of Hurricane Katrina in August 2005. In 2006, the Corps announced “Twelve Actions for Change,” which included a set of actions intended to transform the Corps’ priorities, processes, and planning and apply lessons learned from Hurricanes Katrina and Rita. Among these actions was to employ independent review for projects with significant consequences, especially the potential for loss of life if the project were to fail.

Section 2034 of WRDA of 2007

In November 2007, Congress passed WRDA 2007 and included section 2034, which establishes a 7-year trial period for peer reviews of certain studies of civil works projects; this trial generally applies to project studies initiated by the Corps from November 2005 through November 2014.The Corps was to provide an initial report to Congress on its implementation of the peer review trial under section 2034 by November 2010 and is to provide a final report by November 2013. In February 2011, the Corps submitted its initial report to Congress summarizing its experiences implementing the peer review process in response to the requirement in section 2034 of WRDA 2007.

Section 2034 defines a project study as a feasibility or reevaluation study—including the EIS for that study—or any other study associated with a modification of a water resources project that includes an EIS. Under section 2034, project studies that meet at least one of the following criteria are required to undergo peer review:

The project has an estimated total cost of more than $45 million.

The governor of an affected state requests an independent peer review.

The Chief of Engineers determines that the project study is controversial (i.e., significant public dispute exists as to the project’s size, nature, or effects or its economic or environmental costs or benefits).

In addition, if the head of a federal or state agency charged with reviewing a project study determines the project is likely to have a significant adverse impact on environmental, cultural, or other resources, he or she may request that the Corps consider a peer review by an independent panel of experts.

WRDA 2007 also provides some instances where exceptions may be made to peer review. For example, the Corps may exclude from peer review certain projects having a total estimated cost of more than $45 million but do not include an EIS, have not been determined by the Corps to be controversial, and come below specified thresholds of adverse impacts. The Corps may also exclude other project studies meeting certain exclusion criteria. For example, the Corps may exclude studies that involve only the rehabilitation or replacement of existing hydropower turbines, lock structures, or flood control gates within the same footprint and for the same purpose as an existing water resources project; an activity for which the Corps and industry have ample experience, so the activity may be considered routine; and minimal life safety risk.

Section 2034 also has requirements for the Corps concerning the peer review panel and its independence, as well as for timing the peer review and publishing peer review reports, as described in more detail below.

Contractor Selection, Conflicts of Interest, and Independence

Under section 2034, the Corps is required to contract with NAS, a similar independent scientific and technical advisory organization, or an “eligible organization” to establish a panel of experts that will review a project study. Section 2034 defines an eligible organization as having the following five characteristics: is a 501(c)(3) tax-exempt organization, is free from conflicts of interest, does not carry out or advocate for or against federal water resources has experience in establishing and administering panels.

Section 2034 states that when establishing peer review panels, contractors must apply NAS’s policy for selecting committee members to ensure that they also have no conflicts of interest. The NAS Policy on Committee Composition and Balance and Conflicts of Interest outlines several criteria for selecting peer review panel members, including the following:

All panel members must be highly qualified in terms of knowledge, training, and experience.

The knowledge, experience, and perspectives of the panel members must be thoughtfully and carefully assessed and balanced in terms of the subtleties and complexities of the particular scientific, technical, and other issues to be addressed.

Potential sources of bias must be assessed to determine that the panel’s report will not be compromised by issues of bias or lack of objectivity.

Panel members must not have financial interests that could significantly impair their objectivity or create an unfair competitive advantage for any person or organization.

Panel members must not obtain and use, or intend to use, confidential information not reasonably available to the public for their own or direct and substantial economic benefit.

Panel members must not serve as a member on a peer review panel that is to review the panel member’s own work.

Panel members must not have become committed to a fixed position related to the review for which they have a significant directly related interest or duty.

Persons currently employed by the agency sponsoring the study cannot be panel members, except in extremely limited special circumstances.

Additionally, section 2034 requires that both the experts selected for the peer review panels and the organizations managing the peer review selections be independent. Section 2034 does not define the term independent, but both the 2002 NAS peer review study and OMB’s Final Information Quality Bulletin for Peer Review regard independent to mean external to the Corps. Specifically, the NAS study states that a fully independent review can be accomplished only by reviewers who are free of conflicts of interest and are appointed by a group external to the Corps. Similarly, the OMB bulletin states that independent reviewers are generally not employed by the agency or office producing the document.

Timing of Peer Review and Publishing of Peer Review Reports

Section 2034 requires that the peer review be conducted during the period from the signing of the feasibility cost-share agreement between the Corps and the local sponsor and 60 days after the last day of the public comment period for the draft project study. Additionally, section 2034 lists three points during the feasibility study process at each of which the Chief of Engineers must consider whether to initiate peer review: when the without-project conditions—current and forecasted conditions if the project were not constructed—are identified, when the array of alternatives to be considered are identified, and when the preferred alternative is identified.

Figure 2 shows the key steps in the feasibility study process, including those specified in section 2034. The Corps can conduct peer review at any time during the steps shown highlighted in gray in the figure, but according to Corps officials it generally conducts peer review after the draft feasibility report has been completed. The Washington-level review shown as the final step in the figure concludes with a signed Chief’s report for project studies that will be submitted to Congress for authorization.

Section 2034 also requires the Corps to prepare and make publicly available a written response to all completed peer review reports before it finalizes project studies. The Corps must provide Congress with a copy of both the completed peer review report and the Corps’ written response when the signed Chief’s report or other final decision document for the project study is transmitted to Congress.

Corps Guidance for Implementing Peer Review

The Corps’ Engineering Circular 1165-2-209 (EC 209) was issued in January 2010 and establishes its civil works review policy, which outlines the processes for implementing product review requirements for Corps civil works projects. EC 209 was developed to include the specific requirements for independent peer review contained in section 2034, OMB’s 2004 peer review guidance, as well as other Corps policy considerations. EC 209 requires that districts, in coordination with the relevant Corps planning center of expertise, prepare review plans for project studies. These review plans are to describe the appropriate levels of potential review that the specific project study will be subject to, such as the district’s quality control procedures, agency technical review, peer review, and policy and legal review. If a project study review plan indicates that a peer review will not be conducted, then the district is required to develop a risk-based recommendation for why the peer review is not required. This recommendation should document, among other things, that the project study is of such limited scope or impact that it would not benefit from a peer review.

It Is Unclear How Many Peer Reviews Have Been Completed in Response to Section 2034 Requirements

Since enactment of WRDA 2007, 49 Corps civil works project studies have undergone peer review as of January 2012, but it is unclear how many of these reviews were performed in response to the requirements in section 2034. This is because the Corps does not make specific determinations or track whether a peer review is being conducted in response to the requirements of section 2034. Of the 49 project studies that underwent peer review, the majority were for ecosystem restoration projects, flood risk management projects, or deep draft navigation projects. (App. II lists the 49 project studies that underwent peer review since WRDA 2007 was passed, including information on project and study type, as well as the district, division, and planning center of expertise associated with each study.)

Moreover, it is not possible to determine how many project studies were required to undergo peer review in response to WRDA’s section 2034 requirements because the Corps does not centrally track project studies and could not provide us a list of all project studies that fell within the scope of section 2034. Our review of relevant Corps documents for the 49 project studies that underwent peer review, such as review plans and completed review reports, found that none of these documents specifies the authority under which peer reviews were conducted. Corps headquarters officials told us that the Corps does not make specific determinations as to whether a peer review is being conducted under section 2034 but instead focuses on ensuring that the peer review is being carried out in compliance with EC 209, which, in their view, complies with section 2034. These officials also told us that, to ensure the quality of Corps project studies, the agency may choose to conduct peer reviews under its other authorities, even if those peer reviews are not required by section 2034.

In February 2011, the Corps submitted its initial report to Congress in response to the requirement in section 2034 of WRDA 2007 summarizing its experiences implementing the peer review process. In the report, the Corps noted that the 29 peer reviews that had been completed as of February 2011 followed the procedures described in agency and OMB guidance. The Corps report stated that, in its view, section 2034 provisions reinforce and add further definition to the Corps’ process. Nevertheless, because the Corps did not distinguish which studies had been selected for peer review in accordance with section 2034, we believe that it did not provide Congress with the type of information required by section 2034 that would help congressional decision makers evaluate the trial program.

Completed Peer Reviews Have Cost Millions of Dollars in Direct and Indirect Costs

The 49 peer reviews conducted by the Corps since November 2007 resulted in direct costs of about $9 million in contract costs and contract administration fees. In addition, Corps staff resources were also used to manage peer reviews, but these costs are not fully quantifiable. Furthermore, the addition of peer review to the Corps study process has resulted in indirect costs by altering project study schedules because of the additional time required to complete the peer review. In some cases where a peer review was not planned for during the early stages of the study process, significant delays in the project studies have resulted from the addition of the peer review.

Direct Costs Related to Peer Reviews Have Totaled Over $9 Million

The 49 project studies for which the Corps completed peer reviews since November 2007 cost about $9 million in contract costs and contract administration fees to establish and manage the expert panels for these reviews. In addition, Corps staff resources were also used to manage the peer reviews, but these costs are not fully quantifiable. The Corps used the services of three contractors to manage the peer review process: the nonprofit Battelle Memorial Institute, which managed 46 of the reviews; the nonprofit Noblis, which managed two; and NAS, which managed one. The cost per panel varied considerably. For example, the contracts managed by Battelle cost from about $76,000 to $484,000 for studies that underwent peer review, but the panel managed by NAS cost over $500,000. (See app. II for information on the contract costs for each of the 49 peer reviews.) In addition to the $9 million in contractor costs, the Corps incurred about $109,000 for the administration of the contracts for the 49 peer reviews. Specifically, the Corps used two different entities— the Institute for Water Resources and the Army Research Office—to administer these contracts, and both of these entities charged administration fees.contract cost.

These fees ranged from no fee to 3 percent of the Corps staff resources were also used to manage the peer review process—including developing the scope of work for reviews, coordinating establishment of contracts, reviewing contract proposals, and responding to panel comments that the Corps received during a peer review process. Corps district, division, headquarters, and planning- center-of-expertise staff also spent time managing the peer review process. The total of these costs, however, is not fully quantifiable across all Corps districts because not all districts track district staff time spent on peer reviews. For those districts that did track or could estimate district staff time spent on peer review-related activities, we found the following examples of the staff resources that may have been dedicated to managing and responding to peer review activities:

The Green Bay Dredged Material Management Plan peer review cost about $101,000 in staff time, according to data provided by district officials. But district officials involved in this peer review said that the cost in terms of staff time may have been higher than typical because this peer review was the first conducted for a study in that district.

The Chatfield Water Reallocation Study peer review cost about $20,000 in staff time as of December 2011, but the Corps response has not been completed for this peer review, and additional staff time could be involved.

For the Boston Harbor study, district officials estimated that costs totaled about $77,000 for district staff time, agency technical review team labor, and contractor fees for assisting the district with responding to peer review panel comments.

For the American River study, district officials estimated that costs came to about $40,000 for district staff time.

Similar to district staff time, other staff time involved in managing the peer review process, including headquarters, division, and some planning centers of expertise time, is also not always tracked and therefore not fully quantifiable: some of these positions are funded with general funds and not project-specific funds, according to Corps officials we spoke with. We did find two examples, however, where planning centers of expertise staff time devoted to peer review related activities was tracked. In these two instances, the cost of planning centers of expertise staff time devoted to peer review activities amounted to about $12,000 for the peer review of the Boston Harbor study and about $32,000 for the American River study.

Addition of Peer Review Has Affected Project Study Schedules

The addition of peer review to the Corps study process has also had an indirect cost because it has affected project study schedules. Planning centers of expertise and district officials estimate that obtaining the contract and executing the peer review generally take about a year. The breakdown for the peer review process, according to some of these officials, is about 3 months to initiate the contract; 3 to 6 months for the review to be completed; and an additional 3 months to close out the review, which involves responding to and receiving clarification on panel comments. Some of these processes occur concurrently with other aspects of the project study, but some parts of the peer review process, such as responding to panel comments, may add time to the study schedule. According to Corps officials, the addition of peer review to the project schedule adds steps to the review process and takes time away from other projects. In addition, according to several Corps officials, some project studies have been delayed because the district did not allocate funding for the peer review and therefore had to wait until additional funding was available. In some cases, this delay added significant time to the schedule. In contrast, according to some Corps division and planning- center-of-expertise officials, when the project manager had built in time for the peer review and had identified funding for it early, the peer review process had much less of an impact on the overall project study schedule.

Local sponsors are also concerned about the impact that this additional time is adding to project studies, according to Corps officials and local sponsors we spoke with. Their concern arises largely because local sponsors share the cost of the Corps study and depend on its timely completion. District officials told us that because of the cost-sharing requirements and the current economic environment demand is greater from local sponsors for the Corps to finish studies quickly and keep costs down. Two local sponsors told us that delays negatively affect local sponsors because they can lose business if a project is not completed in a timely manner. Similarly, sponsors are accountable to their own local governments or state legislatures, and additional delays or time required for peer review can create challenges in getting continued support for a project. For example, in the case of the Green Bay Dredged Material Management Plan, Corps officials told us that peer review increased the cost of the project and caused a 5- to 6-month delay at a time when the local sponsor was attempting to acquire grant money contingent on completion of the dredged material management plan.

The Corps’ Process for Initiating Peer Review Is More Expansive and Less Flexible Than Section 2034 Requirements

The Corps’ process for determining whether a project study is subject to peer review is more expansive than section 2034 requirements because it uses broader criteria; this has resulted in peer reviews of studies that are outside the scope of section 2034. In addition, the process the Corps uses does not include the flexibility provided in section 2034 to exclude certain project studies from peer review. Moreover, some studies are undergoing peer review that do not warrant it, according to some Corps officials we spoke with.

The Corps’ Process for Determining the Need for Peer Review Is More Expansive Than Section 2034 Requirements

The Corps’ process for determining whether a project study is subject to peer review uses criteria that are broader than the requirements of section 2034. As table 1 shows, the Corps relies on its guidance outlined in EC 209 when selecting project studies for peer review, and this guidance extends beyond section 2034 requirements.

Consequently, the Corps has selected some studies for review based in part on criteria included in EC 209 that are not required by section 2034, and others that are outside the scope of section 2034. For example, according to our analysis of review plans for 44 peer reviewed project studies, over one-third identified criteria that related to both section 2034 and other authorities. In addition, the Corps process for determining whether a peer review is required has resulted in 30 project studies undergoing peer review that were outside the time parameters identified in section 2034. Based on our analysis of the characteristics of these studies, the Corps’ process was applied to all studies and reports regardless of when they were initiated, whereas section 2034 applies to project studies initiated from November 2005 through November 2014. Specifically, section 2034 applies to (1) project studies initiated from November 2005 through November 2007 and for which the array of alternatives had not been identified, and (2) project studies initiated from November 2007 through November 2014. As a result, over half (30 of 49) of the peer reviews conducted since the enactment of WRDA 2007 were for project studies that did not fall under the scope of section 2034 because the studies were initiated before November 2005.

Another reason the Corps’ process for selecting studies for peer review is more expansive than the scope of section 2034 is that Corps’ guidance does not clearly define “project study.” The guidance refers to a wide range of project studies, decision documents, and work products that may be subject to peer review, whereas section 2034 defines a project study subject to peer review as a feasibility or reevaluation study, including the EIS, or any other study associated with the modification of a water resources project that includes an EIS. According to our analysis of the 49 studies that underwent peer review, some of these studies did not fit this definition. Specifically, 34 of the 49 studies that underwent peer review were feasibility or reevaluation studies which are project studies as defined by section 2034 requirements, 8 were other kinds of reports that included an EIS and therefore may have been subject to section 2034 requirements, and 7 were neither feasibility nor reevaluation studies and did not include an EIS and therefore did not fit the definition of a project study subject to peer review under section 2034. For more details on each of the studies that underwent peer review, see appendix II.

The Corps Does Not Use the Flexibility in Section 2034 to Exclude Studies from Review

The Corps’ process for determining whether peer review is required for project studies does not include the flexibility provided in section 2034 to exclude certain project studies from otherwise mandatory peer review. EC 209 states that most studies should undergo peer review, and the Corps’ process requires that for any decision document to forgo a peer review, an exclusion must be requested and approved by headquarters. In addition, guidance provided to Corps staff on how to implement EC 209 discourages requests for exclusions, noting that time should not be wasted shopping around for exclusion requests. Furthermore, agency guidance and Corps headquarters officials, including the Director of Civil Works, highlight the value and importance of peer review in achieving the agency’s mission, noting that an extra set of eyes is beneficial. In addition, Corps headquarters officials told us, and agency guidance highlights language from the WRDA 2007 conference report, that “ection 2034 permits the Chief of Engineers to exclude a very limited number of project studies from independent peer review.” We believe, however, that the Corps has misconstrued this statement and overstated its significance. This statement is part of the explanation of the exclusion paragraph (a)(5), and does not apply to the provision as a whole; therefore, this statement pertains to how many studies for which peer review is mandatory would be eligible for exclusion.another relevant statement in a House committee report on WRDA 2007 suggests that 26 studies over 7 years, or about 4 studies per year, would be expected to be subject to peer review. Additionally, the 2002 NAS study—which is prominently mentioned throughout the subsequent legislative history of WRDA 2007—states that not all Corps water resources project planning studies will require external, independent review, but the Corps should institute external review for studies that are expensive, that will affect a large area, that are highly controversial, or that involve high levels of risk.

In reviewing the exclusion requests that it receives, Corps headquarters determines whether the studies meet any of the mandatory requirements in EC 209 for undergoing peer review. Specifically, the Corps reviews whether the project has a cost estimate of greater than $45 million, represents a threat to health and safety, is controversial, and has had a request for peer review from a governor or the head of a federal or state agency. If studies do not meet any of these criteria, the Corps generally approves the study for exclusion from peer review. From our review of 50 studies that had requested exclusion from peer review between 2009 and 2011, we found that the Corps had granted an exclusion for 37 studies because they did not meet any of the criteria in EC 209 for studies that must undergo peer review except for one study, which did not fit the definition of a project study in section 2034.

Under section 2034, however, the Corps also has the flexibility to exclude studies from peer review that exceed the $45 million threshold if they: do not have an EIS; are not controversial; are expected to have negligible adverse impacts on scarce or unique cultural, historic, or tribal resources; have no substantial impacts on fish and wildlife species or their habitats; and have no more than negligible impacts on threatened or endangered species or their critical habitat. Similarly, under section 2034, the Corps may exclude studies from peer review that involve (1) only the rehabilitation or replacement of existing hydropower turbines, lock structures, or flood control gates within the same footprint and for the same purpose as an existing water resources project; (2) an activity for which there is ample experience within the Corps and industry to treat the activity as being routine; and (3) minimal risk to human life and safety. Nevertheless, according to our analysis of exclusion request documents and headquarters’ responses to these requests, as of November 2011, the Corps had not granted an exclusion based on any of the flexibilities included in section 2034.

Several Corps officials expressed concerns about the Corps exclusion process. Specifically, some officials told us that they were concerned about the cost and time involved and said that the exclusion of projects that do not meet any of the mandatory criteria should be delegated to the division offices. In their opinion such delegation would help streamline the process. Moreover, some of the studies that underwent or are currently undergoing peer review did not warrant it, according to some Corps officials we spoke with. Specifically, we found the following examples of studies that may not have warranted a peer review:

Two dredged material management plans underwent peer review. For example, the Green Bay Dredged Material Management Plan underwent peer review but is not a project study as defined by section 2034. Officials we spoke with said that such plans should not generally require peer review because any significant impacts would be addressed under NEPA and because the Corps has sufficient expertise in the area of dredging.

The Chacon Creek study in southern Texas underwent peer review but should not have, according to some Corps officials we spoke with. This study was for a project that would remove houses from a floodplain, but officials said it should not require peer review because there are no structural components and it did not exceed the $45 million threshold. Corps headquarters denied the request for exclusion and stated that flood studies warrant peer review because of the nature of the hazard and the need to assess the extent and treatment of risk. Headquarters officials highlighted the importance of assessing and addressing such risks in light of Hurricane Katrina and said that flood studies such as Chacon Creek require peer review because of the importance of assessing and decreasing risks associated with flooding.

The Yuba River General Reevaluation study is undergoing peer review but does not warrant it according to some Corps officials. District officials told us that the study does not warrant peer review because the construction work involved has already been completed, and the purpose of the study is to determine the amount the local sponsor should be reimbursed by the Corps.

Officials from several districts, divisions, and planning centers of expertise we spoke with told us that peer review should be focused on larger and more complex or controversial projects and should not be the default approach. Two Corps officials described the Corps’ peer review policy as a one-size-fits-all approach, and one of these officials stated that it is inflexible and risk averse.

Gaps Exist in the Corps’ Process for Screening Its Contractors Who Are Responsible for Selecting Experts for Peer Review Panels

The Corps has a process to review general information on contractors’ conflicts of interest and independence during its contractor selection process, but it does not have a process for reviewing project-specific information provided by contractors to determine if conflicts of interest and independence exist at the project level. The Corps’ contractors, however, have a process for reviewing the appropriate information related to the conflicts of interest and independence of the experts selected for peer review panels at the project level.

For its initial peer reviews, the Corps relied on Battelle to establish and manage the peer review panels. From 2007 to 2009, Battelle managed 15 independent peer reviews for the Corps. The Corps had identified Battelle as a potential contractor for managing its peer review panels as early as August 2007, when WRDA 2007 was being considered. To ensure that Battelle could meet the section 2034 independence requirements, according to Corps officials, the Director of Civil Works and the Chief of Planning and Policy held discussions with Battelle, and officials from the Corps’ Institute for Water Resources met with Battelle to discuss Battelle’s existing review process and the independent peer review requirements of WRDA 2007. Battelle informed the Corps that it met all WRDA 2007 requirements for an eligible organization, and Battelle identified its existing contract with the Army Research Office as a vehicle for employing Battelle to establish and manage peer review panels under section 2034.

During that time, NAS also conducted one independent peer review for the Corps on the Louisiana Coastal Protection and Restoration Program, which charged the Corps with developing a full range of flood control, coastal restoration, and hurricane protection measures for South Louisiana. According to Corps planning-center-of-expertise officials, because of the extensive scope and breadth of the project, NAS was chosen instead of Battelle to conduct that peer review. But Corps headquarters and planning-center-of-expertise officials told us that, over the course of NAS’ review, they realized that NAS would not be the appropriate organization for reviewing individual projects studies because its process was too time-consuming and expensive. A member of that NAS peer review panel also told us that while he would recommend NAS review for larger projects, in his opinion NAS might not be the appropriate organization for reviewing smaller Corps projects.

In 2009, the Corps sought additional contractors to establish and manage peer review panels and began its contractor selection process by putting out a request for proposals. This solicitation included as contract requirements the section 2034 criteria that the organizations establishing and managing peer review panels be independent and free from conflicts of interest. The Corps received six proposals, including one from Battelle, and each of these proposals was then evaluated by a three-person review panel. The panel chairperson told us that the section 2034 criteria that eligible organizations be independent and free from conflicts of interest were considered as minimum qualifications for screening and selection. As a result of this process, the Corps awarded a contract to Battelle—this contract was in addition to the existing contract Battelle already had with the Corps through the Army Research Office—and one to Noblis. The Corps determines which of the two contractors it will use to manage individual peer reviews on the basis of the contractors’ responses to specific project study scope of work requests, described below.

Although the Corps’ contractor screening and selection process identifies general contractor independence and areas of conflicts of interest, the Corps does not have a process for reviewing the selected contractors’ project-specific independence and freedom from conflicts of interest. For each project study undergoing a peer review, the Corps sends both contractors a “scope of work” document, which describes the project study and lists the required contractor qualifications. These qualifications include independence and freedom from conflicts of interest related to the specific project study being reviewed. In response, the contractors send the Corps their proposals for conducting the peer review, which generally include statements that they are independent and free from project- specific conflicts. Nevertheless, we identified a number of weaknesses in the Corps’ approach for reviewing and corroborating this information, including the following:

The Corps’ planning centers of expertise are expected to review the contractors’ overall proposals, but the Corps does not require the centers to ensure that contractors’ statements of independence within the proposals are reviewed and corroborated for each individual project. Although planning-center-of-expertise officials told us that they review the overall proposals, some of these officials also stated that they did not believe that the statements required review because Corps headquarters had already prescreened the contractors during the initial contractor screening and selection process. Furthermore, the Corps has not provided any guidance to the planning centers of expertise or other Corps offices that specifies how those officials should review the contractors’ project-specific statements at the proposal stage and ensure that they are accurate and that the contractors are in fact independent and free from conflicts of interest. Absent such guidance, the Corps cannot ensure that its contractors are independent and free from conflicts of interest at the project level.

The Corps neither conducts any internal conflicts-of-interest checks nor asks contractors for documentation about potential conflicts of interest so that it can determine whether a conflict exists; rather, the Corps allows the contractors to make that determination on their own. As a result, if the contractors do not provide this information to the Corps, the agency does not have a process for otherwise obtaining this information.

Unlike the Corps’ review of contractor independence, the Corps’ contractors do solicit and review information on panel members’ independence and conflicts of interest at the project level. The contractors gather information about prospective panel members using screening questions developed from the scope of work for each peer review. These questions cover issues described in the NAS policy on committee composition and conflicts of interest, such as financial and employment interests and public statements and positions. The peer review reports from both Battelle and Noblis state that they follow both the OMB guidance on peer review and the NAS policy when selecting panel members. According to contractors and Corps officials, district and planning-center-of-expertise officials review the contractors’ screening questions, as well as the resumes of selected experts, and can provide the contractors with additional information about potential conflicts of interest, such as previous work a particular expert may have done for the Corps. Corps officials told us that the contractors follow up on such information where appropriate, but the contractors and a Corps official we spoke to said that it is the contractors who ultimately select the panel members and ensure their independence.

The Corps Has Adopted Most Peer Review Recommendations, Resulting in Technical Improvements but Generally No Changes in Project Decisions

The Corps has adopted and incorporated most of the peer review recommendations it has received. Adoption of these recommendations has resulted in some technical improvements to project study reports but generally has not changed the Corps’ decisions in selecting preferred project designs. According to some Corps officials we spoke with, this is the result of the review occurring too late in the process to effect a change in decision making.

The Corps Has Adopted and Incorporated Most Peer Review Recommendations, Resulting in Technical Improvements

Of the 49 project studies that have undergone peer review, the Corps has provided a final written response for 17. The Corps has adopted 231 of 274 recommendations, partially adopted 31, and rejected 12 for these 17 peer reviews. Several Corps district officials told us that they make every effort not to reject peer review recommendations and that headquarters has directed them to adopt recommendations whenever possible. In fact, some district officials told us that they felt pressure from headquarters to adopt peer review recommendations even when the recommendations would not affect the study outcome and would be burdensome to implement.

The Corps’ adoption of peer review recommendations has improved the technical quality of its project study reports, according to Corps officials and panel members we spoke with. Corps officials complimented the quality and technical competence of panel members and stated that the panels’ recommendations have been helpful in clarifying and strengthening the arguments presented in the studies. Most of the recommendations either requested that the Corps add to or clarify the study report or stated that the study report did not sufficiently address certain issues. The Corps addressed these issues in almost all instances (193 of 201 recommendations) within its written responses to completed peer review reports.

A smaller number of recommendations addressed the underlying assumptions and inputs to the project studies’ economic, engineering, and environmental analyses. Corps revised portions of its analysis on the basis of these kinds of recommendations. In none of these cases did the Corps indicate that the revised analyses would change the study decisions. In one case, according to Corps documents, the revised analysis served to strengthen arguments in favor of its recommended plan. In response to a recommendation concerning an environmental analysis from the peer review of the Mid-Chesapeake Bay Islands Ecosystem Restoration project study, the Corps conducted additional analyses to justify its calculations of environmental benefits. The Corps reported that the additional analyses led to the determination that the selected plan was appropriate but that by considering the ecosystem impacts of the project in a more detailed fashion, justification of the recommended plan was strengthened.

Specifically, 34 out of the total 910 peer review recommendations indicated a problem with the economic analysis, 24 indicated a problem with the engineering analysis, and 19 indicated a problem with the environmental analysis.

Nevertheless, despite these technical improvements, some Corps officials have questioned the benefit of peer review, given the significant amount of time that district staff have to spend managing the process and responding to recommendations. The process for responding to recommendations begins with district officials drafting a written response, which they provide to the panel. The Corps’ response to the peer review recommendations includes a detailed description of the steps that the Corps has taken or will take to incorporate the recommendations into the project study. The contractor then convenes a teleconference at which district officials discuss the draft response with panel members. After this discussion, the panel members provide written feedback—“backcheck responses”—to the Corps stating whether they agree with the district’s response. The district then finalizes its response to the recommendations and forwards the response to its division office. After its review, the division forwards the response to headquarters, where the response is finalized. The final written response is generally published at the same time as the final decision document for the project study. The time between completion of the peer review report and public availability of Corps’ written response therefore varies greatly depending on the individual project. In one case it was 3 months, while in other cases peer review reports have been completed for more than 3 years without a final response from the Corps having been made public.

For Most Studies, Peer Review Occurs Too Late in the Study Process to Affect Decision Making

Corps officials we spoke with told us that peer review recommendations have generally had no impact on the Corps’ decision making process. These Corps officials were not aware of any project studies for which the study outcome changed as a result of peer review. Corps headquarters officials told us that one reason for the lack of impact of peer review on decision-making is because the Corps’ internal review process is identifying the same issues as peer review. Another reason cited by Corps officials for the lack of impact on decision making is the fact that peer review is occurring at the end of the study process. Peer review generally occurs concurrently with the public comment period for the draft study report, which comes after the preferred design has been selected. As a result, some recommendations about alternatives may not have been implemented because the decision on the preferred design had already been made. Selecting a different preferred design at that stage would require the Corps to revisit an already completed selection analysis and decision. For example, in the peer review report on the Cedar River- Cedar Rapids Iowa Flood Risk Management project study, the review panel recommended that the Corps further investigate one of the non- selected design alternatives, because panel members felt that the alternative might achieve project objectives better than the preferred design. The Corps, however, had already selected its design and decided to proceed. The Corps did not adopt this recommendation, stating that it believed its analysis of alternatives was sound and that there was no reasonable expectation that a more detailed analysis of the alternative would result in finding that it had greater net economic benefits than the preferred design.

In contrast, when the Corps has conducted a peer review earlier in the process, opportunities have arisen for positive impacts on a study decision. For example, the American River Common Features project study peer review was conducted early in the study process. According to Corps division and planning-center-of-expertise officials, they conducted peer review early to obtain external input on defining the problem and to inform decision-making due to the complexity of the project. As a result, the peer review began before the alternative formulation briefing, when the without-project conditions were being identified. By employing this approach, the Corps received feedback from the review panel before selecting the preferred design. The panel’s recommendations included three suggested changes to the Corps’ analyses and model calibrations, which the Corps had time to incorporate before conducting the alternative analysis and selection. According to the contractor that managed the peer review, the panel members involved in the American River Common Features peer review also found the timing of the review to be beneficial and suggested that the Corps conduct peer review earlier for other project studies.

The timing of peer review was also addressed in the 2002 NAS study on peer review. NAS recommended that the Corps initiate peer reviews early enough in the study process so that the review results could be meaningfully incorporated into the study or project design and stated that conducting peer review before selecting a recommended plan is essential if the Corps is to benefit from the review. Corps officials nevertheless told us that they have generally chosen to conduct peer reviews later in the process to minimize effects on project study schedules. Corps headquarters officials noted that, for many studies, peer review occurred late in the process because the studies were under way at the time the Corps began requiring peer review. These officials also noted that it would be challenging to assemble a peer review panel to conduct a review early in the study process and retain the same panel to complete this review at the end of the study. Furthermore, Corps headquarters officials noted that a further challenge is implementing section 2033 WRDA requirements along with section 2034. Section 2033 generally requires the Corps to complete feasibility studies within 2 years. According to Corps officials, there is tension between these requirements and it may be challenging to include peer review throughout the study process without altering project study schedules.

Conclusions

Section 2034 established a trial to look at the cost and impact of conducting peer review for controversial and costly projects over a 7-year period. After the trial period, based on information provided by the Corps, Congress could reconsider whether to retain or revise section 2034 or allow it to lapse. Because the Corps generally does not specify the authority under which peer review was conducted, however, it has not provided Congress with the information needed to evaluate the merits of the section 2034 requirements. In addition, the Corps’ implementation of peer review has not focused on the larger, more complex, and controversial projects that were contemplated when section 2034 was enacted and as recommended by NAS a decade ago. As a result, project studies are being selected to undergo peer review that may not be warranted and may thereby be increasing project costs and schedules needlessly. Further, essential to the integrity of the peer review process is the assurance that the Corps has effective processes not only to ensure overall contractor independence and freedom from conflicts of interest but also to ensure project-level independence and freedom from conflicts of interest. The Corps’ current process, however, has a number of weaknesses with respect to ensuring no conflicts of interest exist at the project level. Finally, with peer review generally occurring late in the Corps’ project study process, peer review serves more to strengthen the Corps’ presentation of its decisions than to influence its decision making. This effect runs counter to what NAS recommended in 2002, that realizing the benefits of peer review requires the results to be used as inputs in the decision-making process. By choosing to apply peer review late in the project study process, the Corps has effectively chosen to not use the results of peer review to enhance its decision-making process and ensure selection of the most effective project alternatives.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers to take the following three actions: To facilitate congressional evaluation of the 7-year trial period outlined in section 2034, the Corps should: Identify for each past and future peer review the specific statutory authority under which the peer review was conducted and the criteria triggering peer review under the Corps’ civil works review policy.

To better reflect section 2034 and provide more effective stewardship of public resources and ensure efficient and effective operations, the Corps should:

Revise the criteria in the Corps’ process for conducting peer review to focus on larger, more complex, and controversial projects; to encourage peer review to occur earlier in the study process; and to include exclusions to peer review that align with section 2034.

Develop a documented process to ensure that contractors are independent and free from conflicts of interest on a project-specific basis.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Defense for review and comment. In its written comments, reprinted in appendix III, the department generally concurred with our recommendations. Specifically, in response to our first recommendation, the department agreed that the Corps should, and stated that it will, identify for each past and future peer review the specific statutory authority under which the peer review was conducted and the criteria triggering peer review under the Corps’ civil works review policy. In response to our second recommendation, the department partially concurred, stating that it agreed that peer review should be focused on studies that will significantly benefit from peer review and that initiating reviews early is advantageous. Nevertheless, the department noted that early involvement must be balanced with having sufficient data and analysis available for review and also highlighted work under way at the agency to overhaul its planning processes, which includes efforts to better align product reviews for greater effectiveness. In response to our third recommendation, the department agreed that the Corps should develop a documented process to ensure that contractors are independent and free from conflicts of interest on a project-specific basis.

Although the department generally concurred with our recommendations, it disagreed with our report’s finding that the Corps’ process does not use the flexibility provided in section 2034, and it disagreed that some studies have undergone review that did not warrant it. The department stated that the Corps has carefully deliberated in support of the agency decision to conduct peer review on the three studies noted in our report and also stated that the Corps stands by all of its decisions to date to grant or deny exclusions from peer review. Nevertheless, the department stated that as part of the Corps’ ongoing review of the civil works review policy, it will assess the effectiveness of its criteria and how the criteria are applied to determine which studies should be considered for exclusion. In addition, the department expressed concern about the level of weight given in the report to anecdotal remarks from field-level officials, who in the department’s opinion may not have had the benefit of the corporate vision supporting the Army Civil Works Program. We disagree with the department’s characterization of our methodology. As clearly described in the scope and methodology section of this report, we interviewed officials who had a corporate-level perspective, as well as those who had a project-level perspective. Specifically, to obtain a corporate-level view, we interviewed senior level officials from Corps headquarters, the Institute for Water Resources, and the planning centers of expertise involved in managing the peer reviews. In addition, to get a project-level perspective and to assess the impact of peer review on division and district offices, we interviewed officials in all of the Corps’ eight divisions, and from 10 geographically dispersed Corps districts that had conducted studies that underwent peer review. We also interviewed the three contractors and selected peer review panel members and local sponsors of Corps civil works projects. We believe that the report provides a balanced perspective from both the headquarters and field levels.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Defense, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and other interested parties. This report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this work were to examine (1) the number of Corps project studies that have undergone independent peer review in response to section 2034 of the Water Resources and Development Act (WRDA) of 2007, (2) the cost of these peer reviews, (3) the extent to which the U.S. Army Corps of Engineers’ (Corps) process for determining if a project study is subject to peer review is consistent with section 2034, (4) the process the Corps uses to ensure that the contractors it hires and the experts the contractors select to review project studies are independent and free from conflicts of interest, and (5) the extent to which recommendations from peer reviews have been incorporated into project studies. We focused on peer reviews for which reports had been completed since WRDA 2007 was enacted.

To address all of these objectives, we reviewed relevant legal requirements, policy guidance, review plans, and peer review reports for project studies that were subject to a peer review and for which a peer review report had been completed since WRDA 2007 was enacted. In addition, we selected a nongeneralizable sample of six peer reviews to examine in greater depth to better understand the costs associated with conducting these reviews, as well as the overall impact of the process on the timeline of the project study and the study outcome. We chose these reviews as illustrative examples and selected one from each of the Corps’ planning centers of expertise and at least one for each of the three contractors the Corps has used to manage peer reviews since enactment of WRDA 2007. Although the information derived from analysis of these case studies cannot be generalized, these examples provide valuable insights into the peer review process. We conducted semistructured interviews with officials from Corps headquarters, the planning centers of expertise involved in managing the peer reviews, all of the Corps’ eight divisions, and from 10 geographically dispersed Corps districts that had conducted studies that underwent peer review. We also conducted semistructured interviews with the three contractors, as well as selected peer review panel members and local sponsors of Corps civil works projects.

To determine the number of studies that have undergone peer review in response to section 2034 of WRDA, we reviewed all completed peer review reports, plus Corps reports and information on completed peer reviews. We reviewed information on completed peer reviews obtained from headquarters, the planning centers of expertise, divisions, and selected districts. We also reviewed information on completed peer reviews obtained from the contractors that established the peer review panels and the entities the Corps used to administer these contracts: the Institute for Water Resources and the Army Research Office.

To determine the cost of these reviews, we reviewed contract award documents and information on contract costs from the contractors. Generally, we relied on the contract award amounts reported in the contracts to determine the cost of the contracts awarded for establishing review panels. For four contract awards, the contract work included establishing a peer review panel and additional work. For these awards, we therefore relied on information provided by the contractor on the portion of the contract cost that was for the peer review. For the contract award for peer review of a local sponsor-led study, we relied on information from the local sponsor and the contractor on the cost of the award. In addition, for the six case study peer reviews, we analyzed information on costs associated with managing the review process, including cost data and estimates provided by districts with regard to district and other staff time involved in peer review. In cases where we reported cost data including staff time associated with completing peer review, we asked knowledgeable officials about the data system and the quality of the data and determined that they were sufficiently reliable for our purposes. In cases where we reported estimates of these costs, we asked officials about how these estimates were developed and determined that they were sufficiently reliable for our purposes.

To determine the extent to which the Corps’ process for determining if a study is subject to peer review is consistent with section 2034, we analyzed the legal requirements and relevant policy guidance for determining when to conduct peer review. We also reviewed documentation on decisions to conduct peer review included in review plans and documents requesting exclusion from peer review. In addition, we reviewed information on the characteristics of studies that underwent peer review, including date initiated, whether an environmental impact statement was included, and the type of study. We identified this information in review plans, study drafts, signed Chief’s reports, and other Corps study-related documents; Corps officials from relevant districts or divisions reviewed this information.

To determine the process the Corps uses to ensure that the contractors it hires and the experts the contractors select are independent and free from conflicts of interest, we reviewed information on contractor selection obtained from Corps headquarters and the Institute for Water Resources. We also reviewed documentation from the contractors that outlined contractor and reviewer qualifications, as well as the National Academy of Sciences Policy on Committee Composition and Balance and Conflicts of Interest.

To determine the extent to which peer review recommendations are incorporated into project studies, we reviewed information obtained from headquarters, the planning centers of expertise, divisions, and selected districts on how the Corps responds to peer review recommendations. We also reviewed all peer review recommendations contained in completed peer review reports, as well as all responses to peer review recommendations contained in the Corps’ published responses to the completed peer review reports.

We conducted this performance audit from April 2011 to March 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Characteristics of Studies That Underwent Peer Review

Forty-six of the 49 peer reviews completed in table 2 below were conducted by Battelle Memorial Institute; Noblis completed the Green Bay Dredged Material Management Plan and the Wood River Levee System General Reevaluation Report peer reviews, and the National Academy of Sciences conducted the Louisiana Coastal Protection and Restoration peer review. As table 2 shows, the studies that underwent peer review came under the areas of ecosystem restoration (19 of 49), flood risk management (15 of 49), deep draft navigation (7 of 49), coastal storm damage reduction (5 of 49), inland navigation (2 of 49), and water management and reallocation (1 of 49). According to our analysis of Corps documents, 32 of the 49 studies included an environmental impact statement (EIS), 19 of 49 were initiated after November 2005, and 42 of 49 had an estimated total project cost greater than $45 million.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual listed above, Vondalee R. Hunt, Assistant Director; Darnita Akers; Elizabeth Beardsley; and Janice Ceperich made significant contributions to this report. Ellen Wo Chu, Cindy Gilbert, Richard P. Johnson, Ben Shouse, and Kiki Theodoropoulos also made key contributions.