Army Corps of Engineers: Peer Review Process for Civil Works Project Studies Can Be Improved
Published: 2012-03-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Section 2034 of the Water Resources Development Act of 2007 requires that certain U.S. Army Corps of Engineers (Corps) civil works project studies undergo independent external peer review to assess the adequacy and acceptability of the methods, models, and analyses used. In the act, Congress established a 7-year trial period for this requirement and also required the Corps to submit two reports on its experiences with the peer review process.
GAO was asked to examine (1) the number of Corps project studies that have undergone independent peer review in response to section 2034, (2) the cost of these peer reviews, (3) the extent to which the Corps process for determining if a project study is subject to peer review is consistent with section 2034, (4) the process the Corps uses to ensure that the contractors it hires and the experts the contractors select to review project studies are independent and free from conflicts of interest, and (5) the extent to which peer review recommendations have been incorporated into project studies. GAO reviewed relevant laws, agency guidance, and documents and interviewed Corps officials and contractors.
section_title: What GAO Found
paragraphs: Since enactment of the Water Resources Development Act of 2007, 49 project studies have undergone peer review but it is unclear how many were performed in response to section 2034 requirements because the Army Corps of Engineers (Corps) does not make specific determinations or track if a peer review is being conducted under section 2034. In February 2011, in response to section 2034, the Corps submitted its initial report to Congress summarizing its implementation of the peer review process. In its report, however, the Corps did not distinguish which studies had been selected for peer review in accordance with section 2034 and therefore, did not provide Congress information that would help decision makers evaluate the requirements of section 2034 at the end of the trial period.
The 49 peer reviews resulted in both direct and indirect costs. Specifically, these peer reviews resulted in direct costs of over $9 million in contract costs and fees. In addition, Corps staff resources were used to manage the reviews, although these costs are not fully quantifiable. Furthermore, the addition of peer review to the Corps study process has resulted in indirect costs by altering project study schedules to allow for time needed to complete peer reviews. In some cases where a peer review was not planned during the early stages of the study process, significant delays to project studies occurred while funds were sought to pay for the peer review. In contrast, according to some Corps officials, when project managers have built in time and identified funding for peer reviews early, the process has had less of an impact on project study schedules.
The Corps process for determining whether a project study is subject to peer review is more expansive than section 2034 requirements because it uses broader criteria, resulting in peer reviews of studies outside the scope of section 2034. In addition, the process the Corps uses does not include the flexibility provided in section 2034, which allows for the exclusion of certain project studies from peer review. Moreover, some studies are undergoing peer reviews that do not warrant it, according to some Corps officials GAO spoke with.
The Corps has a process to review general information on contractors conflicts of interest and independence when selecting them to establish peer review panels, but it does not have a process for reviewing project-level information on conflicts of interest and independence. As a result, it cannot be assured that contractors do not have conflicts at the project-level. In contrast, the Corps contractors do have a process for reviewing information related to conflicts of interest and the independence of experts selected for each peer review panel.
The Corps has adopted and incorporated into its project study reports most of the peer review recommendations it has received. Doing so has resulted in some technical improvements to study reports but generally has not changed the Corps decisions about project alternatives, in part because the peer review process occurs too late in the project study process to affect decision making, according to some Corps officials GAO spoke with. As a result, some recommendations about alternatives may not have been implemented because the decision on the preferred design had already been made.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Department of Defense direct the Corps to, among other actions, better track peer review studies, revise the criteria for determining which studies undergo peer review and the timing of these reviews, and improve its process for ensuring contractor independence. The department generally concurred with these recommendations.

section_title: Letter
paragraphs: Through its civil works program, the U.S. Army Corps of Engineers  (Corps) constructs, operates, and maintains thousands of civil works  projects related to water resources across the United States. These  projects aim to provide safe and reliable waterways; reduce risk to  people, homes, and communities from flooding and coastal storms;  restore and protect the environment; and address water resources  challenges. A Corps civil works project generally starts with a study of a  water resources issue and the development of various alternatives to  address it. Such studies can span the full range of Corps civil works  projects, and can include those that are small and low impact and others  that are large and complex, with potentially significant economic and  environmental impacts. Through its civil works program, the Corps  operates 50 centers of expertise and seven research laboratories that  assist its eight divisions and 38 district offices in the planning, design, and  technical review of civil works projects. Six of these centers are focused  on the quality and effectiveness of water resources planning and are  referred to as “planning centers of expertise.”
Through its civil works projects, the Corps provides vital public  engineering services in peace and war to strengthen the nation’s security,  energize the economy, and reduce risks from disasters. These projects  involve navigation and flood control activities, environmental restoration,  and emergency response—most recently including emergency response  to Missouri River flooding and rebuilding after Hurricane Katrina.  Technical errors in past studies of Corps projects, however, had raised  concerns about the effectiveness of the Corps’ internal review processes  and the quality of the studies that the Corps used as a basis for its civil  works projects. For example, in March 2006, we reported that certain  studies completed by the Corps from 1992 through 2002 were fraught  with errors, mistakes, and miscalculations and used invalid assumptions  and outdated data.understated costs, overstated benefits, and did not provide a reasonable  basis for decision making. Similar findings have been documented in  reviews by the National Academy of Sciences (NAS) and other  organizations, which concluded that the Corps’ review processes needed  to be strengthened.
We also reported that these Corps studies  In the wake of these reports, Congress passed section 2034 of the Water  Resources Development Act (WRDA) of 2007, which requires that studies  for certain Corps projects undergo independent peer review. To conduct  such peer review, the Corps hires a contractor to select a panel of  independent experts, who assess the adequacy and acceptability of the  economic, engineering, and environmental methods, models, and  analyses used in a Corps’ project study. Upon completion of the peer  review, the Corps is to consider recommendations from the review before  making a final decision on the project.
You asked us to review the efforts the Corps has made to implement and  comply with the independent peer review requirements in section 2034 of  WRDA 2007. This report examines (1) the number of Corps project  studies that have undergone independent external peer review in  response to section 2034, (2) the cost of these peer reviews, (3) the  extent to which the Corps’ process for determining if a project study is  subject to peer review is consistent with section 2034, (4) the process the  Corps uses to ensure that the contractors it hires and the experts the  contractors select to review project studies are independent and free from  conflicts of interest, and (5) the extent to which peer review  recommendations have been incorporated into project studies.
To conduct this work, we reviewed relevant legal requirements, policy  guidance, review plans, and peer review reports for project studies that  were subject to a peer review and for which a report had been completed  since WRDA 2007 was passed. In addition, we selected a  nongeneralizable sample of six peer reviews to examine in greater depth,  to better understand the costs associated with conducting these reviews,  as well as the overall impact of the process on the timeline of the project  study and the study’s outcome. Because this sample was a nonprobability  sample, the information derived from these reviews is not generalizable to  all peer reviews, but the reviews serve as illustrative examples that  provide valuable insights into the Corps’ peer review process; we  selected a review from each planning center of expertise and at least one  for each of the three contractors selected to conduct peer reviews. We  conducted semistructured interviews with officials from Corps  headquarters, the planning centers of expertise involved in managing the  peer reviews, all of the Corps’ eight divisions, and from 10 geographically  dispersed districts that had conducted project studies that underwent peer  review. We also conducted semistructured interviews with the three  contractors, as well as with selected peer review panel members and  local sponsors of Corps civil works projects.
To determine the number of studies that underwent peer review and the  cost of these reviews, we reviewed all completed peer review reports and  contract award documentation. In addition, for the six peer reviews we  examined in depth, we analyzed information on costs associated with  managing the review process, including those associated with district and  other staff time. To determine the extent to which the Corps’ process for  determining if a study is subject to peer review is consistent with section  2034, we analyzed the legal requirements and relevant Corps policy  guidance for determining when to conduct peer reviews, and we reviewed  documentation on Corps decisions. To determine the process the Corps  uses to ensure that the contractors it hires and the experts the contractors  select are independent and free from conflicts of interest, we analyzed  relevant Corps policy guidance and reviewed documentation provided by  the contractors to demonstrate that they and the selected experts meet  the requirements. To determine the extent to which recommendations are  incorporated into project studies, we analyzed relevant policy guidance  and summarized peer review recommendations and Corps responses,  but we did not assess the quality of the peer review recommendations or  the technical sufficiency of the Corps responses to these  recommendations. In addition, for the six reviews examined in depth, we  analyzed project studies, recommendations, and Corps responses to  describe the overall timeline and impact of the peer review process.  Appendix I describes our scope and methodology in greater detail.
We conducted this performance audit from April 2011 to March 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Corps Process for Civil Works Project Studies
paragraphs: The purpose of the Corps’ civil works project study process is to inform  federal decision makers whether a water resources project warrants  further federal investment. The study process is conducted in two phases:  reconnaissance and feasibility. In the reconnaissance phase, the Corps  conducts an initial evaluation of potential solutions to a water resources  problem. If the Corps determines that a project potentially warrants  federal investment, it proceeds to a more detailed feasibility study. The  feasibility phase generally begins with the signing of a feasibility cost- share agreement between the Corps and the local project sponsor.
Feasibility studies are generally prepared by the Corps’ 38 district offices,  with review and oversight provided by the cognizant Corps division office  and by headquarters. During the feasibility phase, the Corps formulates  and evaluates alternative plans for achieving the project’s objectives and  reviews the proposed project to assess whether the benefits of  constructing it outweigh its costs. At the beginning of this phase, a  feasibility scoping meeting is held to bring the Corps, the local sponsor,  and other agencies together to reach agreement on the problems and  solutions to be investigated during the feasibility study and the scope of  the analysis required. The next step includes an alternative formulation  briefing to identify and resolve any legal or policy concerns and to obtain  headquarters approval of the tentatively selected plan and to release the  draft report to the public. Finally, the draft feasibility report—which  presents the study results and findings, including those developed in the  reconnaissance phase—is released to the public. At the conclusion of the  feasibility phase, the Corps selects a recommended plan for proceeding  with the project.
The feasibility report also includes analysis and documentation to meet  the requirements of the National Environmental Policy Act (NEPA).  Under NEPA, federal agencies are to assess the effects of major federal  actions, such as Corps construction projects, that significantly affect the  environment and prepare a detailed statement on the environmental  impacts of those actions. NEPA has two principal purposes: (1) to ensure  that an agency carefully considers detailed information concerning  significant environmental impacts and (2) to ensure that this information  will be made available to the public. NEPA requires an agency to prepare  a detailed statement on the environmental impacts of any “major federal  action” significantly affecting the environment. NEPA implementing  regulations generally require an agency to prepare either an  environmental assessment or an environmental impact statement  (EIS).procedures—such as providing the public with an opportunity to comment  on the draft EIS for at least 45 days.
NEPA implementing regulations also specify requirements and  Corps project studies have historically been subject to various levels of  internal and external review under a number of authorities as well as the  Chief of Engineers’ responsibility to ensure the quality of Corps studies.  For example, in 1902, Congress created the Board of Engineers for  Rivers and Harbors, which was the result of efforts to address  inconsistent treatment of proposed Corps projects. The board was made  up of Corps staff. Until 1992, when Congress terminated the board, it  reviewed thousands of Corps studies for civil works projects and made  unfavorable recommendations on more than half. At the time the board  was abolished, there was concern that too much duplicative review was  occurring between the board and other internal Corps review processes.  Subsequently, in the Flood Control Act of 1944, Congress established a  mechanism for external review of Corps projects by giving the head of the  Department of the Interior and the governors of affected states an  opportunity to comment on proposed Corps projects before authorization.  Furthermore, starting in 1970, under NEPA, environmental impact  statements for Corps projects were required to be sent to the heads of  other federal agencies and governors of affected states for comment. As  a result of the Flood Control Act of 1970, Congress created the position of  Assistant Secretary of the Army for Civil Works, who coordinates the  review of Corps studies with the Office of Management and Budget  (OMB) before they are submitted to Congress.
section_title: Increasing Interest in Peer Review of Corps Project Studies
paragraphs: Recent congressional interest in establishing an independent external  peer review process for Corps project studies began in the late 1990s,  following a series of damaging reports and events, including allegations  that the Corps had manipulated information to justify projects.  Investigations conducted by NAS and the Army’s Inspector General  identified various problems with the Corps internal review process,  including the manipulation of economic analysis and potential institutional  bias toward large construction projects. Around this time, WRDA 2000  required the Corps to contract with NAS to study and make  recommendations concerning the use of peer review for feasibility  reports, including recommending potential criteria to determine how to  apply peer review. In 2002, NAS released its study concluding that the  Corps’ more complex water resources project planning studies should be  subject to external, independent review. The study also found that not  all Corps project studies necessarily require such review, recommending  instead that external peer review be reserved for studies that are  expensive, will affect a large area, are highly controversial, or involve high  levels of risk. The study estimated that about five Corps projects per year  would likely be subject to this level of review. According to the NAS study,  criteria for selecting the appropriate level of review should balance the  risks and consequences of inadequate review against the resources  required for more complex and stringent levels of review. In addition, the  study identified several criteria that should be considered in determining  the appropriate level of review for Corps studies, primarily that as project  magnitude and risks increase, an increasing degree of independence and  scope of review are warranted (see fig. 1).
In addition to recommendations related to the appropriate level of review  for project studies, the 2002 NAS study made several other  recommendations about the Corps’ peer review process. It recommended  that peer review results be presented to the Chief of Engineers before a  final decision on a project study is made, that the Chief of Engineers  respond in writing to each key point of the peer review report, and that  peer review be initiated early enough in the Corps’ study process so that  review results can be meaningfully incorporated into project design.
After NAS published its 2002 study, OMB in December 2004 issued its  Final Information Quality Bulletin for Peer Review citing the Information   as well as its general authorities to oversee the quality of  Quality Act,agency information, analyses, and regulatory actions. This OMB bulletin  established governmentwide guidance on enhancing peer review  practices and covers what information is subject to peer review, the  selection of appropriate peer reviewers, opportunities for public  participation, and related issues. The Corps’ Engineering Circular 1105- 2-408 (EC 408) was issued in May 2005 and established procedures for  ensuring the credibility and quality of Corps documents by supplementing  its previous review process, including to add external peer review to its  review process in special cases where risk and magnitude warrant this  level of review.
The Corps faced further criticism after the failure of Corps levees and  floodwalls in New Orleans in the wake of Hurricane Katrina in August  2005. In 2006, the Corps announced “Twelve Actions for Change,” which  included a set of actions intended to transform the Corps’ priorities,  processes, and planning and apply lessons learned from Hurricanes  Katrina and Rita. Among these actions was to employ independent review  for projects with significant consequences, especially the potential for loss  of life if the project were to fail.
section_title: Section 2034 of WRDA of 2007
paragraphs: In November 2007, Congress passed WRDA 2007 and included section  2034, which establishes a 7-year trial period for peer reviews of certain  studies of civil works projects; this trial generally applies to project studies  initiated by the Corps from November 2005 through November 2014.The Corps was to provide an initial report to Congress on its  implementation of the peer review trial under section 2034 by November  2010 and is to provide a final report by November 2013. In February  2011, the Corps submitted its initial report to Congress summarizing its  experiences implementing the peer review process in response to the  requirement in section 2034 of WRDA 2007.
Section 2034 defines a project study as a feasibility or reevaluation  study—including the EIS for that study—or any other study associated  with a modification of a water resources project that includes an EIS.  Under section 2034, project studies that meet at least one of the following  criteria are required to undergo peer review:
The project has an estimated total cost of more than $45 million.
The governor of an affected state requests an independent peer  review.
The Chief of Engineers determines that the project study is  controversial (i.e., significant public dispute exists as to the project’s  size, nature, or effects or its economic or environmental costs or  benefits).
In addition, if the head of a federal or state agency charged with reviewing  a project study determines the project is likely to have a significant  adverse impact on environmental, cultural, or other resources, he or she  may request that the Corps consider a peer review by an independent  panel of experts.
WRDA 2007 also provides some instances where exceptions may be  made to peer review. For example, the Corps may exclude from peer  review certain projects having a total estimated cost of more than  $45 million but do not include an EIS, have not been determined by the  Corps to be controversial, and come below specified thresholds of  adverse impacts. The Corps may also exclude other project studies  meeting certain exclusion criteria. For example, the Corps may exclude  studies that involve only the rehabilitation or replacement of existing hydropower turbines,  lock structures, or flood control gates within the same footprint and for  the same purpose as an existing water resources project; an activity for which the Corps and industry have ample experience,  so the activity may be considered routine; and minimal life safety risk.
Section 2034 also has requirements for the Corps concerning the peer  review panel and its independence, as well as for timing the peer review  and publishing peer review reports, as described in more detail below.
section_title: Contractor Selection, Conflicts of Interest, and Independence
paragraphs: Under section 2034, the Corps is required to contract with NAS, a similar  independent scientific and technical advisory organization, or an “eligible  organization” to establish a panel of experts that will review a project  study. Section 2034 defines an eligible organization as having the  following five characteristics:  is a 501(c)(3) tax-exempt organization, is free from conflicts of interest, does not carry out or advocate for or against federal water resources has experience in establishing and administering panels.
Section 2034 states that when establishing peer review panels,  contractors must apply NAS’s policy for selecting committee members to  ensure that they also have no conflicts of interest. The NAS Policy on  Committee Composition and Balance and Conflicts of Interest outlines  several criteria for selecting peer review panel members, including the  following:
All panel members must be highly qualified in terms of knowledge,  training, and experience.
The knowledge, experience, and perspectives of the panel members  must be thoughtfully and carefully assessed and balanced in terms of  the subtleties and complexities of the particular scientific, technical,  and other issues to be addressed.
Potential sources of bias must be assessed to determine that the  panel’s report will not be compromised by issues of bias or lack of  objectivity.
Panel members must not have financial interests that could  significantly impair their objectivity or create an unfair competitive  advantage for any person or organization.
Panel members must not obtain and use, or intend to use, confidential  information not reasonably available to the public for their own or  direct and substantial economic benefit.
Panel members must not serve as a member on a peer review panel  that is to review the panel member’s own work.
Panel members must not have become committed to a fixed position  related to the review for which they have a significant directly related  interest or duty.
Persons currently employed by the agency sponsoring the study  cannot be panel members, except in extremely limited special  circumstances.
Additionally, section 2034 requires that both the experts selected for the  peer review panels and the organizations managing the peer review  selections be independent. Section 2034 does not define the term  independent, but both the 2002 NAS peer review study and OMB’s Final  Information Quality Bulletin for Peer Review regard independent to mean  external to the Corps. Specifically, the NAS study states that a fully independent review can be accomplished only by reviewers who are free  of conflicts of interest and are appointed by a group external to the Corps.  Similarly, the OMB bulletin states that independent reviewers are  generally not employed by the agency or office producing the document.
section_title: Timing of Peer Review and Publishing of Peer Review Reports
paragraphs: Section 2034 requires that the peer review be conducted during the  period from the signing of the feasibility cost-share agreement between  the Corps and the local sponsor and 60 days after the last day of the  public comment period for the draft project study. Additionally, section  2034 lists three points during the feasibility study process at each of  which the Chief of Engineers must consider whether to initiate peer  review: when the without-project conditions—current and forecasted  conditions if the project were not constructed—are identified, when the array of alternatives to be considered are identified, and when the preferred alternative is identified.
Figure 2 shows the key steps in the feasibility study process, including  those specified in section 2034. The Corps can conduct peer review at  any time during the steps shown highlighted in gray in the figure, but  according to Corps officials it generally conducts peer review after the  draft feasibility report has been completed. The Washington-level review  shown as the final step in the figure concludes with a signed Chief’s  report for project studies that will be submitted to Congress for  authorization.
Section 2034 also requires the Corps to prepare and make publicly  available a written response to all completed peer review reports before it  finalizes project studies. The Corps must provide Congress with a copy of  both the completed peer review report and the Corps’ written response  when the signed Chief’s report or other final decision document for the  project study is transmitted to Congress.
section_title: Corps Guidance for Implementing Peer Review
paragraphs: The Corps’ Engineering Circular 1165-2-209 (EC 209) was issued in  January 2010 and establishes its civil works review policy, which outlines  the processes for implementing product review requirements for Corps  civil works projects. EC 209 was developed to include the specific  requirements for independent peer review contained in section 2034,  OMB’s 2004 peer review guidance, as well as other Corps policy  considerations. EC 209 requires that districts, in coordination with the  relevant Corps planning center of expertise, prepare review plans for  project studies. These review plans are to describe the appropriate levels  of potential review that the specific project study will be subject to, such  as the district’s quality control procedures, agency technical review, peer  review, and policy and legal review. If a project study review plan  indicates that a peer review will not be conducted, then the district is  required to develop a risk-based recommendation for why the peer review  is not required. This recommendation should document, among other  things, that the project study is of such limited scope or impact that it  would not benefit from a peer review.
section_title: It Is Unclear How Many Peer Reviews Have Been Completed in Response to Section 2034 Requirements
paragraphs: Since enactment of WRDA 2007, 49 Corps civil works project studies  have undergone peer review as of January 2012, but it is unclear how  many of these reviews were performed in response to the requirements in  section 2034. This is because the Corps does not make specific  determinations or track whether a peer review is being conducted in  response to the requirements of section 2034. Of the 49 project studies  that underwent peer review, the majority were for ecosystem restoration  projects, flood risk management projects, or deep draft navigation  projects. (App. II lists the 49 project studies that underwent peer review  since WRDA 2007 was passed, including information on project and  study type, as well as the district, division, and planning center of  expertise associated with each study.)
Moreover, it is not possible to determine how many project studies were  required to undergo peer review in response to WRDA’s section 2034  requirements because the Corps does not centrally track project studies  and could not provide us a list of all project studies that fell within the  scope of section 2034. Our review of relevant Corps documents for the 49  project studies that underwent peer review, such as review plans and  completed review reports, found that none of these documents specifies  the authority under which peer reviews were conducted. Corps  headquarters officials told us that the Corps does not make specific  determinations as to whether a peer review is being conducted under  section 2034 but instead focuses on ensuring that the peer review is  being carried out in compliance with EC 209, which, in their view,  complies with section 2034. These officials also told us that, to ensure the  quality of Corps project studies, the agency may choose to conduct peer  reviews under its other authorities, even if those peer reviews are not  required by section 2034.
In February 2011, the Corps submitted its initial report to Congress in  response to the requirement in section 2034 of WRDA 2007 summarizing  its experiences implementing the peer review process. In the report, the  Corps noted that the 29 peer reviews that had been completed as of  February 2011 followed the procedures described in agency and OMB  guidance. The Corps report stated that, in its view, section 2034  provisions reinforce and add further definition to the Corps’ process.  Nevertheless, because the Corps did not distinguish which studies had  been selected for peer review in accordance with section 2034, we  believe that it did not provide Congress with the type of information  required by section 2034 that would help congressional decision makers  evaluate the trial program.
section_title: Completed Peer Reviews Have Cost Millions of Dollars in Direct and Indirect Costs
paragraphs: The 49 peer reviews conducted by the Corps since November 2007  resulted in direct costs of about $9 million in contract costs and contract  administration fees. In addition, Corps staff resources were also used to  manage peer reviews, but these costs are not fully quantifiable.  Furthermore, the addition of peer review to the Corps study process has  resulted in indirect costs by altering project study schedules because of  the additional time required to complete the peer review. In some cases  where a peer review was not planned for during the early stages of the  study process, significant delays in the project studies have resulted from  the addition of the peer review.
subsections: section_title: Direct Costs Related to Peer Reviews Have Totaled Over $9 Million
paragraphs: The 49 project studies for which the Corps completed peer reviews since  November 2007 cost about $9 million in contract costs and contract  administration fees to establish and manage the expert panels for these  reviews. In addition, Corps staff resources were also used to manage the  peer reviews, but these costs are not fully quantifiable. The Corps used  the services of three contractors to manage the peer review process: the  nonprofit Battelle Memorial Institute, which managed 46 of the reviews;  the nonprofit Noblis, which managed two; and NAS, which managed one.  The cost per panel varied considerably. For example, the contracts  managed by Battelle cost from about $76,000 to $484,000 for studies that  underwent peer review, but the panel managed by NAS cost over  $500,000. (See app. II for information on the contract costs for each of  the 49 peer reviews.) In addition to the $9 million in contractor costs, the  Corps incurred about $109,000 for the administration of the contracts for  the 49 peer reviews. Specifically, the Corps used two different entities— the Institute for Water Resources and the Army Research Office—to  administer these contracts, and both of these entities charged  administration fees.contract cost.
These fees ranged from no fee to 3 percent of the  Corps staff resources were also used to manage the peer review  process—including developing the scope of work for reviews,  coordinating establishment of contracts, reviewing contract proposals,  and responding to panel comments that the Corps received during a peer  review process. Corps district, division, headquarters, and planning- center-of-expertise staff also spent time managing the peer review  process. The total of these costs, however, is not fully quantifiable across  all Corps districts because not all districts track district staff time spent on  peer reviews. For those districts that did track or could estimate district  staff time spent on peer review-related activities, we found the following  examples of the staff resources that may have been dedicated to  managing and responding to peer review activities:
The Green Bay Dredged Material Management Plan peer review cost  about $101,000 in staff time, according to data provided by district  officials. But district officials involved in this peer review said that the  cost in terms of staff time may have been higher than typical because  this peer review was the first conducted for a study in that district.
The Chatfield Water Reallocation Study peer review cost about  $20,000 in staff time as of December 2011, but the Corps response  has not been completed for this peer review, and additional staff time  could be involved.
For the Boston Harbor study, district officials estimated that costs  totaled about $77,000 for district staff time, agency technical review  team labor, and contractor fees for assisting the district with  responding to peer review panel comments.
For the American River study, district officials estimated that costs  came to about $40,000 for district staff time.
Similar to district staff time, other staff time involved in managing the peer  review process, including headquarters, division, and some planning  centers of expertise time, is also not always tracked and therefore not  fully quantifiable: some of these positions are funded with general funds  and not project-specific funds, according to Corps officials we spoke with.  We did find two examples, however, where planning centers of expertise  staff time devoted to peer review related activities was tracked. In these  two instances, the cost of planning centers of expertise staff time devoted  to peer review activities amounted to about $12,000 for the peer review of  the Boston Harbor study and about $32,000 for the American River study.
section_title: Addition of Peer Review Has Affected Project Study Schedules
paragraphs: The addition of peer review to the Corps study process has also had an  indirect cost because it has affected project study schedules. Planning  centers of expertise and district officials estimate that obtaining the  contract and executing the peer review generally take about a year. The  breakdown for the peer review process, according to some of these  officials, is about 3 months to initiate the contract; 3 to 6 months for the  review to be completed; and an additional 3 months to close out the  review, which involves responding to and receiving clarification on panel  comments. Some of these processes occur concurrently with other  aspects of the project study, but some parts of the peer review process,  such as responding to panel comments, may add time to the study  schedule. According to Corps officials, the addition of peer review to the  project schedule adds steps to the review process and takes time away  from other projects. In addition, according to several Corps officials, some  project studies have been delayed because the district did not allocate  funding for the peer review and therefore had to wait until additional  funding was available. In some cases, this delay added significant time to  the schedule. In contrast, according to some Corps division and planning- center-of-expertise officials, when the project manager had built in time  for the peer review and had identified funding for it early, the peer review  process had much less of an impact on the overall project study  schedule.
Local sponsors are also concerned about the impact that this additional  time is adding to project studies, according to Corps officials and local  sponsors we spoke with. Their concern arises largely because local  sponsors share the cost of the Corps study and depend on its timely  completion. District officials told us that because of the cost-sharing  requirements and the current economic environment demand is greater  from local sponsors for the Corps to finish studies quickly and keep costs  down. Two local sponsors told us that delays negatively affect local  sponsors because they can lose business if a project is not completed in  a timely manner. Similarly, sponsors are accountable to their own local  governments or state legislatures, and additional delays or time required  for peer review can create challenges in getting continued support for a  project. For example, in the case of the Green Bay Dredged Material  Management Plan, Corps officials told us that peer review increased the  cost of the project and caused a 5- to 6-month delay at a time when the  local sponsor was attempting to acquire grant money contingent on  completion of the dredged material management plan.
section_title: The Corps’ Process for Initiating Peer Review Is More Expansive and Less Flexible Than Section 2034 Requirements
paragraphs: The Corps’ process for determining whether a project study is subject to  peer review is more expansive than section 2034 requirements because it  uses broader criteria; this has resulted in peer reviews of studies that are  outside the scope of section 2034. In addition, the process the Corps  uses does not include the flexibility provided in section 2034 to exclude  certain project studies from peer review. Moreover, some studies are  undergoing peer review that do not warrant it, according to some Corps  officials we spoke with.
subsections: section_title: The Corps’ Process for Determining the Need for Peer Review Is More Expansive Than Section 2034 Requirements
paragraphs: The Corps’ process for determining whether a project study is subject to  peer review uses criteria that are broader than the requirements of  section 2034. As table 1 shows, the Corps relies on its guidance outlined  in EC 209 when selecting project studies for peer review, and this  guidance extends beyond section 2034 requirements.
Consequently, the Corps has selected some studies for review based in  part on criteria included in EC 209 that are not required by section 2034,  and others that are outside the scope of section 2034. For example,  according to our analysis of review plans for 44 peer reviewed project  studies, over one-third identified criteria that related to both section  2034 and other authorities. In addition, the Corps process for determining  whether a peer review is required has resulted in 30 project studies  undergoing peer review that were outside the time parameters identified  in section 2034. Based on our analysis of the characteristics of these  studies, the Corps’ process was applied to all studies and reports  regardless of when they were initiated, whereas section 2034 applies to  project studies initiated from November 2005 through November 2014.  Specifically, section 2034 applies to (1) project studies initiated from  November 2005 through November 2007 and for which the array of  alternatives had not been identified, and (2) project studies initiated from  November 2007 through November 2014. As a result, over half (30 of 49)  of the peer reviews conducted since the enactment of WRDA 2007 were  for project studies that did not fall under the scope of section 2034  because the studies were initiated before November 2005.
Another reason the Corps’ process for selecting studies for peer review is  more expansive than the scope of section 2034 is that Corps’ guidance  does not clearly define “project study.” The guidance refers to a wide  range of project studies, decision documents, and work products that may  be subject to peer review, whereas section 2034 defines a project study  subject to peer review as a feasibility or reevaluation study, including the  EIS, or any other study associated with the modification of a water  resources project that includes an EIS. According to our analysis of the  49 studies that underwent peer review, some of these studies did not fit  this definition. Specifically, 34 of the 49 studies that underwent peer  review were feasibility or reevaluation studies which are project studies as  defined by section 2034 requirements, 8 were other kinds of reports that  included an EIS and therefore may have been subject to section 2034  requirements, and 7 were neither feasibility nor reevaluation studies and  did not include an EIS and therefore did not fit the definition of a project  study subject to peer review under section 2034. For more details on  each of the studies that underwent peer review, see appendix II.
section_title: The Corps Does Not Use the Flexibility in Section 2034 to Exclude Studies from Review
paragraphs: The Corps’ process for determining whether peer review is required for  project studies does not include the flexibility provided in section 2034 to  exclude certain project studies from otherwise mandatory peer review. EC  209 states that most studies should undergo peer review, and the Corps’  process requires that for any decision document to forgo a peer review,  an exclusion must be requested and approved by headquarters. In  addition, guidance provided to Corps staff on how to implement EC 209  discourages requests for exclusions, noting that time should not be  wasted shopping around for exclusion requests. Furthermore, agency  guidance and Corps headquarters officials, including the Director of Civil  Works, highlight the value and importance of peer review in achieving the  agency’s mission, noting that an extra set of eyes is beneficial. In  addition, Corps headquarters officials told us, and agency guidance  highlights language from the WRDA 2007 conference report, that  “ection 2034 permits the Chief of Engineers to exclude a very limited  number of project studies from independent peer review.” We believe,  however, that the Corps has misconstrued this statement and overstated  its significance. This statement is part of the explanation of the exclusion  paragraph (a)(5), and does not apply to the provision as a whole;  therefore, this statement pertains to how many studies for which peer  review is mandatory would be eligible for exclusion.another relevant statement in a House committee report on WRDA 2007  suggests that 26 studies over 7 years, or about 4 studies per year, would  be expected to be subject to peer review. Additionally, the 2002 NAS  study—which is prominently mentioned throughout the subsequent  legislative history of WRDA 2007—states that not all Corps water  resources project planning studies will require external, independent  review, but the Corps should institute external review for studies that are  expensive, that will affect a large area, that are highly controversial, or  that involve high levels of risk.
In reviewing the exclusion requests that it receives, Corps headquarters  determines whether the studies meet any of the mandatory requirements  in EC 209 for undergoing peer review. Specifically, the Corps reviews  whether the project has a cost estimate of greater than $45 million,  represents a threat to health and safety, is controversial, and has had a  request for peer review from a governor or the head of a federal or state  agency. If studies do not meet any of these criteria, the Corps generally  approves the study for exclusion from peer review. From our review of 50  studies that had requested exclusion from peer review between 2009 and  2011, we found that the Corps had granted an exclusion for 37 studies  because they did not meet any of the criteria in EC 209 for studies that  must undergo peer review except for one study, which did not fit the  definition of a project study in section 2034.
Under section 2034, however, the Corps also has the flexibility to exclude  studies from peer review that exceed the $45 million threshold if they: do  not have an EIS; are not controversial; are expected to have negligible  adverse impacts on scarce or unique cultural, historic, or tribal resources;  have no substantial impacts on fish and wildlife species or their habitats;  and have no more than negligible impacts on threatened or endangered  species or their critical habitat. Similarly, under section 2034, the Corps  may exclude studies from peer review that involve (1) only the  rehabilitation or replacement of existing hydropower turbines, lock  structures, or flood control gates within the same footprint and for the  same purpose as an existing water resources project; (2) an activity for  which there is ample experience within the Corps and industry to treat the  activity as being routine; and (3) minimal risk to human life and safety.  Nevertheless, according to our analysis of exclusion request documents  and headquarters’ responses to these requests, as of November 2011,  the Corps had not granted an exclusion based on any of the flexibilities  included in section 2034.
Several Corps officials expressed concerns about the Corps exclusion  process. Specifically, some officials told us that they were concerned  about the cost and time involved and said that the exclusion of projects  that do not meet any of the mandatory criteria should be delegated to the  division offices. In their opinion such delegation would help streamline the  process. Moreover, some of the studies that underwent or are currently  undergoing peer review did not warrant it, according to some Corps  officials we spoke with. Specifically, we found the following examples of  studies that may not have warranted a peer review:
Two dredged material management plans underwent peer review. For  example, the Green Bay Dredged Material Management Plan  underwent peer review but is not a project study as defined by section  2034. Officials we spoke with said that such plans should not  generally require peer review because any significant impacts would  be addressed under NEPA and because the Corps has sufficient  expertise in the area of dredging.
The Chacon Creek study in southern Texas underwent peer review  but should not have, according to some Corps officials we spoke with.  This study was for a project that would remove houses from a  floodplain, but officials said it should not require peer review because  there are no structural components and it did not exceed the $45  million threshold. Corps headquarters denied the request for exclusion  and stated that flood studies warrant peer review because of the  nature of the hazard and the need to assess the extent and treatment  of risk. Headquarters officials highlighted the importance of assessing  and addressing such risks in light of Hurricane Katrina and said that  flood studies such as Chacon Creek require peer review because of  the importance of assessing and decreasing risks associated with  flooding.
The Yuba River General Reevaluation study is undergoing peer  review but does not warrant it according to some Corps officials.  District officials told us that the study does not warrant peer review  because the construction work involved has already been completed,  and the purpose of the study is to determine the amount the local  sponsor should be reimbursed by the Corps.
Officials from several districts, divisions, and planning centers of expertise  we spoke with told us that peer review should be focused on larger and  more complex or controversial projects and should not be the default  approach. Two Corps officials described the Corps’ peer review policy as  a one-size-fits-all approach, and one of these officials stated that it is  inflexible and risk averse.
section_title: Gaps Exist in the Corps’ Process for Screening Its Contractors Who Are Responsible for Selecting Experts for Peer Review Panels
paragraphs: The Corps has a process to review general information on contractors’  conflicts of interest and independence during its contractor selection  process, but it does not have a process for reviewing project-specific  information provided by contractors to determine if conflicts of interest  and independence exist at the project level. The Corps’ contractors,  however, have a process for reviewing the appropriate information related  to the conflicts of interest and independence of the experts selected for  peer review panels at the project level.
For its initial peer reviews, the Corps relied on Battelle to establish and  manage the peer review panels. From 2007 to 2009, Battelle managed 15  independent peer reviews for the Corps. The Corps had identified Battelle  as a potential contractor for managing its peer review panels as early as  August 2007, when WRDA 2007 was being considered. To ensure that  Battelle could meet the section 2034 independence requirements,  according to Corps officials, the Director of Civil Works and the Chief of  Planning and Policy held discussions with Battelle, and officials from the  Corps’ Institute for Water Resources met with Battelle to discuss  Battelle’s existing review process and the independent peer review  requirements of WRDA 2007. Battelle informed the Corps that it met all  WRDA 2007 requirements for an eligible organization, and Battelle  identified its existing contract with the Army Research Office as a vehicle  for employing Battelle to establish and manage peer review panels under  section 2034.
During that time, NAS also conducted one independent peer review for  the Corps on the Louisiana Coastal Protection and Restoration Program,  which charged the Corps with developing a full range of flood control,  coastal restoration, and hurricane protection measures for South  Louisiana. According to Corps planning-center-of-expertise officials,  because of the extensive scope and breadth of the project, NAS was  chosen instead of Battelle to conduct that peer review. But Corps  headquarters and planning-center-of-expertise officials told us that, over  the course of NAS’ review, they realized that NAS would not be the  appropriate organization for reviewing individual projects studies because  its process was too time-consuming and expensive. A member of that  NAS peer review panel also told us that while he would recommend NAS  review for larger projects, in his opinion NAS might not be the appropriate  organization for reviewing smaller Corps projects.
In 2009, the Corps sought additional contractors to establish and manage  peer review panels and began its contractor selection process by putting  out a request for proposals. This solicitation included as contract  requirements the section 2034 criteria that the organizations establishing  and managing peer review panels be independent and free from conflicts  of interest. The Corps received six proposals, including one from Battelle,  and each of these proposals was then evaluated by a three-person  review panel. The panel chairperson told us that the section 2034 criteria  that eligible organizations be independent and free from conflicts of  interest were considered as minimum qualifications for screening and  selection. As a result of this process, the Corps awarded a contract to  Battelle—this contract was in addition to the existing contract Battelle  already had with the Corps through the Army Research Office—and one  to Noblis. The Corps determines which of the two contractors it will use  to manage individual peer reviews on the basis of the contractors’  responses to specific project study scope of work requests, described  below.
Although the Corps’ contractor screening and selection process identifies  general contractor independence and areas of conflicts of interest, the  Corps does not have a process for reviewing the selected contractors’  project-specific independence and freedom from conflicts of interest. For  each project study undergoing a peer review, the Corps sends both  contractors a “scope of work” document, which describes the project  study and lists the required contractor qualifications. These qualifications  include independence and freedom from conflicts of interest related to the  specific project study being reviewed. In response, the contractors send  the Corps their proposals for conducting the peer review, which generally  include statements that they are independent and free from project- specific conflicts. Nevertheless, we identified a number of weaknesses  in the Corps’ approach for reviewing and corroborating this information,  including the following:
The Corps’ planning centers of expertise are expected to review the  contractors’ overall proposals, but the Corps does not require the  centers to ensure that contractors’ statements of independence within  the proposals are reviewed and corroborated for each individual  project. Although planning-center-of-expertise officials told us that  they review the overall proposals, some of these officials also stated  that they did not believe that the statements required review because  Corps headquarters had already prescreened the contractors during  the initial contractor screening and selection process. Furthermore,  the Corps has not provided any guidance to the planning centers of  expertise or other Corps offices that specifies how those officials  should review the contractors’ project-specific statements at the  proposal stage and ensure that they are accurate and that the  contractors are in fact independent and free from conflicts of interest.  Absent such guidance, the Corps cannot ensure that its contractors  are independent and free from conflicts of interest at the project level.
The Corps neither conducts any internal conflicts-of-interest checks  nor asks contractors for documentation about potential conflicts of  interest so that it can determine whether a conflict exists; rather, the  Corps allows the contractors to make that determination on their own.  As a result, if the contractors do not provide this information to the  Corps, the agency does not have a process for otherwise obtaining  this information.
Unlike the Corps’ review of contractor independence, the Corps’  contractors do solicit and review information on panel members’  independence and conflicts of interest at the project level. The contractors  gather information about prospective panel members using screening  questions developed from the scope of work for each peer review. These  questions cover issues described in the NAS policy on committee  composition and conflicts of interest, such as financial and employment  interests and public statements and positions. The peer review reports  from both Battelle and Noblis state that they follow both the OMB  guidance on peer review and the NAS policy when selecting panel  members. According to contractors and Corps officials, district and  planning-center-of-expertise officials review the contractors’ screening  questions, as well as the resumes of selected experts, and can provide  the contractors with additional information about potential conflicts of  interest, such as previous work a particular expert may have done for the  Corps. Corps officials told us that the contractors follow up on such  information where appropriate, but the contractors and a Corps official we  spoke to said that it is the contractors who ultimately select the panel  members and ensure their independence.
section_title: The Corps Has Adopted Most Peer Review Recommendations, Resulting in Technical Improvements but Generally No Changes in Project Decisions
paragraphs: The Corps has adopted and incorporated most of the peer review  recommendations it has received. Adoption of these recommendations  has resulted in some technical improvements to project study reports but  generally has not changed the Corps’ decisions in selecting preferred  project designs. According to some Corps officials we spoke with, this is  the result of the review occurring too late in the process to effect a  change in decision making.
subsections: section_title: The Corps Has Adopted and Incorporated Most Peer Review Recommendations, Resulting in Technical Improvements
paragraphs: Of the 49 project studies that have undergone peer review, the Corps has  provided a final written response for 17. The Corps has adopted 231 of  274 recommendations, partially adopted 31, and rejected 12 for these 17  peer reviews. Several Corps district officials told us that they make every  effort not to reject peer review recommendations and that headquarters  has directed them to adopt recommendations whenever possible. In fact,  some district officials told us that they felt pressure from headquarters to  adopt peer review recommendations even when the recommendations  would not affect the study outcome and would be burdensome to  implement.
The Corps’ adoption of peer review recommendations has improved the  technical quality of its project study reports, according to Corps officials  and panel members we spoke with. Corps officials complimented the  quality and technical competence of panel members and stated that the  panels’ recommendations have been helpful in clarifying and  strengthening the arguments presented in the studies. Most of the  recommendations either requested that the Corps add to or clarify the  study report or stated that the study report did not sufficiently address  certain issues. The Corps addressed these issues in almost all instances  (193 of 201 recommendations) within its written responses to completed  peer review reports.
A smaller number of recommendations addressed the underlying  assumptions and inputs to the project studies’ economic, engineering,  and environmental analyses. Corps revised portions of its analysis on the basis of these kinds of  recommendations. In none of these cases did the Corps indicate that the  revised analyses would change the study decisions. In one case,  according to Corps documents, the revised analysis served to strengthen  arguments in favor of its recommended plan. In response to a  recommendation concerning an environmental analysis from the peer  review of the Mid-Chesapeake Bay Islands Ecosystem Restoration  project study, the Corps conducted additional analyses to justify its  calculations of environmental benefits. The Corps reported that the  additional analyses led to the determination that the selected plan was  appropriate but that by considering the ecosystem impacts of the project  in a more detailed fashion, justification of the recommended plan was  strengthened.
Specifically, 34 out of the total 910 peer review recommendations indicated a problem  with the economic analysis, 24 indicated a problem with the engineering analysis, and 19  indicated a problem with the environmental analysis.
Nevertheless, despite these technical improvements, some Corps officials  have questioned the benefit of peer review, given the significant amount  of time that district staff have to spend managing the process and  responding to recommendations. The process for responding to  recommendations begins with district officials drafting a written response,  which they provide to the panel. The Corps’ response to the peer review  recommendations includes a detailed description of the steps that the  Corps has taken or will take to incorporate the recommendations into the  project study. The contractor then convenes a teleconference at which  district officials discuss the draft response with panel members. After this  discussion, the panel members provide written feedback—“backcheck  responses”—to the Corps stating whether they agree with the district’s  response. The district then finalizes its response to the  recommendations and forwards the response to its division office. After its  review, the division forwards the response to headquarters, where the  response is finalized. The final written response is generally published at  the same time as the final decision document for the project study. The  time between completion of the peer review report and public availability  of Corps’ written response therefore varies greatly depending on the  individual project. In one case it was 3 months, while in other cases peer  review reports have been completed for more than 3 years without a final  response from the Corps having been made public.
section_title: For Most Studies, Peer Review Occurs Too Late in the Study Process to Affect Decision Making
paragraphs: Corps officials we spoke with told us that peer review recommendations  have generally had no impact on the Corps’ decision making process.  These Corps officials were not aware of any project studies for which the  study outcome changed as a result of peer review. Corps headquarters  officials told us that one reason for the lack of impact of peer review on  decision-making is because the Corps’ internal review process is  identifying the same issues as peer review. Another reason cited by  Corps officials for the lack of impact on decision making is the fact that  peer review is occurring at the end of the study process. Peer review  generally occurs concurrently with the public comment period for the draft  study report, which comes after the preferred design has been selected.  As a result, some recommendations about alternatives may not have  been implemented because the decision on the preferred design had  already been made. Selecting a different preferred design at that stage  would require the Corps to revisit an already completed selection analysis  and decision. For example, in the peer review report on the Cedar River- Cedar Rapids Iowa Flood Risk Management project study, the review  panel recommended that the Corps further investigate one of the non- selected design alternatives, because panel members felt that the  alternative might achieve project objectives better than the preferred  design. The Corps, however, had already selected its design and decided  to proceed. The Corps did not adopt this recommendation, stating that it  believed its analysis of alternatives was sound and that there was no  reasonable expectation that a more detailed analysis of the alternative  would result in finding that it had greater net economic benefits than the  preferred design.
In contrast, when the Corps has conducted a peer review earlier in the  process, opportunities have arisen for positive impacts on a study  decision. For example, the American River Common Features project  study peer review was conducted early in the study process. According to  Corps division and planning-center-of-expertise officials, they conducted  peer review early to obtain external input on defining the problem and to  inform decision-making due to the complexity of the project. As a result,  the peer review began before the alternative formulation briefing, when  the without-project conditions were being identified. By employing this  approach, the Corps received feedback from the review panel before  selecting the preferred design. The panel’s recommendations included  three suggested changes to the Corps’ analyses and model calibrations,  which the Corps had time to incorporate before conducting the alternative  analysis and selection. According to the contractor that managed the peer  review, the panel members involved in the American River Common  Features peer review also found the timing of the review to be beneficial  and suggested that the Corps conduct peer review earlier for other project  studies.
The timing of peer review was also addressed in the 2002 NAS study on  peer review. NAS recommended that the Corps initiate peer reviews early  enough in the study process so that the review results could be  meaningfully incorporated into the study or project design and stated that  conducting peer review before selecting a recommended plan is essential  if the Corps is to benefit from the review. Corps officials nevertheless told  us that they have generally chosen to conduct peer reviews later in the  process to minimize effects on project study schedules. Corps  headquarters officials noted that, for many studies, peer review occurred  late in the process because the studies were under way at the time the  Corps began requiring peer review. These officials also noted that it  would be challenging to assemble a peer review panel to conduct a  review early in the study process and retain the same panel to complete  this review at the end of the study. Furthermore, Corps headquarters  officials noted that a further challenge is implementing section 2033  WRDA requirements along with section 2034. Section 2033 generally  requires the Corps to complete feasibility studies within 2 years.  According to Corps officials, there is tension between these requirements  and it may be challenging to include peer review throughout the study  process without altering project study schedules.
section_title: Conclusions
paragraphs: Section 2034 established a trial to look at the cost and impact of  conducting peer review for controversial and costly projects over a 7-year  period. After the trial period, based on information provided by the Corps,  Congress could reconsider whether to retain or revise section 2034 or  allow it to lapse. Because the Corps generally does not specify the  authority under which peer review was conducted, however, it has not  provided Congress with the information needed to evaluate the merits of  the section 2034 requirements. In addition, the Corps’ implementation of  peer review has not focused on the larger, more complex, and  controversial projects that were contemplated when section 2034 was  enacted and as recommended by NAS a decade ago. As a result, project  studies are being selected to undergo peer review that may not be  warranted and may thereby be increasing project costs and schedules  needlessly. Further, essential to the integrity of the peer review process is  the assurance that the Corps has effective processes not only to ensure  overall contractor independence and freedom from conflicts of interest but  also to ensure project-level independence and freedom from conflicts of  interest. The Corps’ current process, however, has a number of  weaknesses with respect to ensuring no conflicts of interest exist at the  project level. Finally, with peer review generally occurring late in the  Corps’ project study process, peer review serves more to strengthen the  Corps’ presentation of its decisions than to influence its decision making.  This effect runs counter to what NAS recommended in 2002, that  realizing the benefits of peer review requires the results to be used as  inputs in the decision-making process. By choosing to apply peer review  late in the project study process, the Corps has effectively chosen to not  use the results of peer review to enhance its decision-making process  and ensure selection of the most effective project alternatives.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the Chief of  Engineers and Commanding General of the U.S. Army Corps of  Engineers to take the following three actions:  To facilitate congressional evaluation of the 7-year trial period outlined in  section 2034, the Corps should:  Identify for each past and future peer review the specific statutory  authority under which the peer review was conducted and the criteria  triggering peer review under the Corps’ civil works review policy.
To better reflect section 2034 and provide more effective stewardship of  public resources and ensure efficient and effective operations, the Corps  should:
Revise the criteria in the Corps’ process for conducting peer review to  focus on larger, more complex, and controversial projects; to  encourage peer review to occur earlier in the study process; and to  include exclusions to peer review that align with section 2034.
Develop a documented process to ensure that contractors are  independent and free from conflicts of interest on a project-specific  basis.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Defense for review  and comment. In its written comments, reprinted in appendix III, the  department generally concurred with our recommendations. Specifically,  in response to our first recommendation, the department agreed that the  Corps should, and stated that it will, identify for each past and future peer  review the specific statutory authority under which the peer review was  conducted and the criteria triggering peer review under the Corps’ civil  works review policy. In response to our second recommendation, the  department partially concurred, stating that it agreed that peer review  should be focused on studies that will significantly benefit from peer  review and that initiating reviews early is advantageous. Nevertheless,  the department noted that early involvement must be balanced with  having sufficient data and analysis available for review and also  highlighted work under way at the agency to overhaul its planning  processes, which includes efforts to better align product reviews for  greater effectiveness. In response to our third recommendation, the  department agreed that the Corps should develop a documented process  to ensure that contractors are independent and free from conflicts of  interest on a project-specific basis.
Although the department generally concurred with our recommendations,  it disagreed with our report’s finding that the Corps’ process does not use  the flexibility provided in section 2034, and it disagreed that some studies  have undergone review that did not warrant it. The department stated that  the Corps has carefully deliberated in support of the agency decision to  conduct peer review on the three studies noted in our report and also  stated that the Corps stands by all of its decisions to date to grant or deny  exclusions from peer review. Nevertheless, the department stated that as  part of the Corps’ ongoing review of the civil works review policy, it will  assess the effectiveness of its criteria and how the criteria are applied to  determine which studies should be considered for exclusion. In addition,  the department expressed concern about the level of weight given in the  report to anecdotal remarks from field-level officials, who in the  department’s opinion may not have had the benefit of the corporate vision  supporting the Army Civil Works Program. We disagree with the  department’s characterization of our methodology. As clearly described in  the scope and methodology section of this report, we interviewed officials  who had a corporate-level perspective, as well as those who had a  project-level perspective. Specifically, to obtain a corporate-level view, we  interviewed senior level officials from Corps headquarters, the Institute for  Water Resources, and the planning centers of expertise involved in  managing the peer reviews. In addition, to get a project-level perspective  and to assess the impact of peer review on division and district offices,  we interviewed officials in all of the Corps’ eight divisions, and from 10  geographically dispersed Corps districts that had conducted studies that  underwent peer review. We also interviewed the three contractors and  selected peer review panel members and local sponsors of Corps civil  works projects. We believe that the report provides a balanced  perspective from both the headquarters and field levels.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Defense, the  Chief of Engineers and Commanding General of the U.S. Army Corps of  Engineers, and other interested parties. This report will be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or mittala@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives for this work were to examine (1) the number of Corps  project studies that have undergone independent peer review in response  to section 2034 of the Water Resources and Development Act (WRDA) of  2007, (2) the cost of these peer reviews, (3) the extent to which the U.S.  Army Corps of Engineers’ (Corps) process for determining if a project  study is subject to peer review is consistent with section 2034, (4) the  process the Corps uses to ensure that the contractors it hires and the  experts the contractors select to review project studies are independent  and free from conflicts of interest, and (5) the extent to which  recommendations from peer reviews have been incorporated into project  studies. We focused on peer reviews for which reports had been  completed since WRDA 2007 was enacted.
To address all of these objectives, we reviewed relevant legal  requirements, policy guidance, review plans, and peer review reports for  project studies that were subject to a peer review and for which a peer  review report had been completed since WRDA 2007 was enacted. In  addition, we selected a nongeneralizable sample of six peer reviews to  examine in greater depth to better understand the costs associated with  conducting these reviews, as well as the overall impact of the process on  the timeline of the project study and the study outcome. We chose these  reviews as illustrative examples and selected one from each of the Corps’  planning centers of expertise and at least one for each of the three  contractors the Corps has used to manage peer reviews since enactment  of WRDA 2007. Although the information derived from analysis of these  case studies cannot be generalized, these examples provide valuable  insights into the peer review process. We conducted semistructured  interviews with officials from Corps headquarters, the planning centers of  expertise involved in managing the peer reviews, all of the Corps’ eight  divisions, and from 10 geographically dispersed Corps districts that had  conducted studies that underwent peer review. We also conducted  semistructured interviews with the three contractors, as well as selected  peer review panel members and local sponsors of Corps civil works  projects.
To determine the number of studies that have undergone peer review in  response to section 2034 of WRDA, we reviewed all completed peer  review reports, plus Corps reports and information on completed peer  reviews. We reviewed information on completed peer reviews obtained  from headquarters, the planning centers of expertise, divisions, and  selected districts. We also reviewed information on completed peer  reviews obtained from the contractors that established the peer review  panels and the entities the Corps used to administer these contracts: the  Institute for Water Resources and the Army Research Office.
To determine the cost of these reviews, we reviewed contract award  documents and information on contract costs from the contractors.  Generally, we relied on the contract award amounts reported in the  contracts to determine the cost of the contracts awarded for establishing  review panels. For four contract awards, the contract work included  establishing a peer review panel and additional work. For these awards,  we therefore relied on information provided by the contractor on the  portion of the contract cost that was for the peer review. For the contract  award for peer review of a local sponsor-led study, we relied on  information from the local sponsor and the contractor on the cost of the  award. In addition, for the six case study peer reviews, we analyzed  information on costs associated with managing the review process,  including cost data and estimates provided by districts with regard to  district and other staff time involved in peer review. In cases where we  reported cost data including staff time associated with completing peer  review, we asked knowledgeable officials about the data system and the  quality of the data and determined that they were sufficiently reliable for  our purposes. In cases where we reported estimates of these costs, we  asked officials about how these estimates were developed and  determined that they were sufficiently reliable for our purposes.
To determine the extent to which the Corps’ process for determining if a  study is subject to peer review is consistent with section 2034, we  analyzed the legal requirements and relevant policy guidance for  determining when to conduct peer review. We also reviewed  documentation on decisions to conduct peer review included in review  plans and documents requesting exclusion from peer review. In addition,  we reviewed information on the characteristics of studies that underwent  peer review, including date initiated, whether an environmental impact  statement was included, and the type of study. We identified this  information in review plans, study drafts, signed Chief’s reports, and other  Corps study-related documents; Corps officials from relevant districts or  divisions reviewed this information.
To determine the process the Corps uses to ensure that the contractors it  hires and the experts the contractors select are independent and free  from conflicts of interest, we reviewed information on contractor selection  obtained from Corps headquarters and the Institute for Water Resources.  We also reviewed documentation from the contractors that outlined  contractor and reviewer qualifications, as well as the National Academy of  Sciences Policy on Committee Composition and Balance and Conflicts of  Interest.
To determine the extent to which peer review recommendations are  incorporated into project studies, we reviewed information obtained from  headquarters, the planning centers of expertise, divisions, and selected  districts on how the Corps responds to peer review recommendations.  We also reviewed all peer review recommendations contained in  completed peer review reports, as well as all responses to peer review  recommendations contained in the Corps’ published responses to the  completed peer review reports.
We conducted this performance audit from April 2011 to March 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Characteristics of Studies That Underwent Peer Review
paragraphs: Forty-six of the 49 peer reviews completed in table 2 below were  conducted by Battelle Memorial Institute; Noblis completed the Green Bay  Dredged Material Management Plan and the Wood River Levee System  General Reevaluation Report peer reviews, and the National Academy of  Sciences conducted the Louisiana Coastal Protection and Restoration  peer review. As table 2 shows, the studies that underwent peer review  came under the areas of ecosystem restoration (19 of 49), flood risk  management (15 of 49), deep draft navigation (7 of 49), coastal storm  damage reduction (5 of 49), inland navigation (2 of 49), and water  management and reallocation (1 of 49). According to our analysis of  Corps documents, 32 of the 49 studies included an environmental impact  statement (EIS), 19 of 49 were initiated after November 2005, and 42 of  49 had an estimated total project cost greater than $45 million.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual listed above, Vondalee R. Hunt, Assistant  Director; Darnita Akers; Elizabeth Beardsley; and Janice Ceperich made  significant contributions to this report. Ellen Wo Chu, Cindy Gilbert,  Richard P. Johnson, Ben Shouse, and Kiki Theodoropoulos also made  key contributions.