Army Corps of Engineers: Known Performance Issues with New Orleans Drainage Canal Pumps Have Been Addressed, but Guidance on Future Contracts Is Needed
Published: 2007-12-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Hurricane Katrina caused several breaches in the floodwalls along three drainage canals in New Orleans, contributing to catastrophic flooding. To restore the pre-Katrina level of hurricane-related flood protection, the Army Corps of Engineers (Corps) decided to acquire several large-capacity pumping systems. During the process of acquiring, testing, and installing the pumping systems, issues with the pump contract and operation of the pumping systems came to light, including several identified in a Corps Independent Team Report (ITR). GAO was asked to evaluate the Corps' efforts to (1) develop contract specifications and award the contract, (2) address pumping system performance issues, (3) document contract modifications, and (4) reconcile contract payments. GAO reviewed contract and testing documents, observed the operation of the pumping system, and interviewed officials from the Corps, its consultants and contractors, and the ITR team.
section_title: What GAO Found
paragraphs: Schedule concerns drove the Corps' decisions in developing specifications for the pumping systems and awarding the contract, but the rush to award the contract resulted in deficiencies in key contract provisions. Specifically, the original factory test requirements were ambiguous, there were only limited provisions for on-site testing, and there were no criteria for acceptance of the pumping systems by the government. The Corps conducted an expedited competition to contract for the pumping systems and selected a supplier for contract award based largely on its ability to deliver the pumping systems by the June 1 start of the 2006 Atlantic hurricane season. The Corps and the contractors have addressed and corrected known performance issues with the pumping systems. Concerns included hydraulic motor vibrations, the design of the hydraulic intake line, suspect pipe welds, and lower than expected pumping capacity. The pumping systems were installed prior to correcting these issues because the Corps believed it was better to have some pumping capacity along the drainage canals during the 2006 hurricane season rather than none, despite uncertainty over how much of the pumping system capacity would be available, and for how long, if needed. Between November 2006 and September 2007, the Corps and the contractors completed all of the repairs and reinstalled the pumping systems. Documents that GAO reviewed indicate that, as of September 2007, each pumping system had been successfully tested on site for at least 2 hours, thus providing greater assurance that they will perform as designed. The contract files for the pumping systems contained the required documentation for the type and value of the contract and associated modifications, though, in a number of cases, documentation was inserted in the contract files several months after modifications were issued and only after the ITR reported its findings. While the ITR correctly noted the absence of some required documentation, GAO found that much of the specific documentation cited as missing was not required for the modifications in question because of the nature and value of these modifications. In addition, while the ITR found that it appeared as though the contractor developed the scope of work and pricing for some of the modifications without a subsequent analysis by the Corps, GAO found no instance of this occurring. As of October 31, 2007, the Corps had paid the contractor about $30.5 million of the $33 million contract amount. In a few instances, the Corps made duplicate payments to the contractor. GAO found that these payments were due to Corps mistakes, not inappropriate billing by the contractor. GAO found no other cases of duplicate payments. The Corps plans to adjust for the duplicate payments by deducting the balance from remaining funds, including any incentive payments, owed to the contractor. According to Corps officials, final payment and reconciliation of the contract is expected by early 2008; however, it is unknown to what extent contract or pump performance issues will affect the final amount paid for the contract during the close-out process.

section_title: Letter
paragraphs: The greater New Orleans metropolitan area sits in the tidal lowlands of  The greater New Orleans metropolitan area sits in the tidal lowlands of  Lake Pontchartrain and is bordered generally on its southern side by the  Lake Pontchartrain and is bordered generally on its southern side by the  Mississippi River and the Gulf of Mexico. Due to the city’s location and  Mississippi River and the Gulf of Mexico. Due to the city’s location and  elevation, which averages six feet below sea level, the area has historically  elevation, which averages six feet below sea level, the area has historically  experienced many floods. In an effort to reduce such floods, a series of  experienced many floods. In an effort to reduce such floods, a series of  levees and other flood control structures have been built over the years to  levees and other flood control structures have been built over the years to  reduce that threat. However, hurricane-induced storm surges, waves, and  reduce that threat. However, hurricane-induced storm surges, waves, and  rainfall continue to pose a threat to New Orleans. To avoid flooding in  rainfall continue to pose a threat to New Orleans. To avoid flooding in  New Orleans from a rain storm, the city’s Sewerage and Water Board  New Orleans from a rain storm, the city’s Sewerage and Water Board  pumps rainwater from the city into three drainage canals located at 17th  pumps rainwater from the city into three drainage canals located at 17th  Street, London Avenue, and Orleans Avenue, which then flows  Street, London Avenue, and Orleans Avenue, which then flows  unrestricted into Lake Pontchartrain. While critical to prevent flooding  unrestricted into Lake Pontchartrain. While critical to prevent flooding  from rainfall, these canals are vulnerable to storm surge from Lake  from rainfall, these canals are vulnerable to storm surge from Lake  Pontchartrain during hurricanes; consequently, floodwalls have been  Pontchartrain during hurricanes; consequently, floodwalls have been  erected along both sides of the canals to protect against storm surge  erected along both sides of the canals to protect against storm surge  overtopping the canals and flooding the city.  overtopping the canals and flooding the city.
On August 29, 2005, storm surge from Hurricane Katrina caused several  On August 29, 2005, storm surge from Hurricane Katrina caused several  breaches in the floodwalls along the 17th Street and London Avenue  breaches in the floodwalls along the 17th Street and London Avenue  canals, contributing to catastrophic flooding in New Orleans. In its efforts  canals, contributing to catastrophic flooding in New Orleans. In its efforts  to restore pre-Katrina levels of hurricane protection to New Orleans by the  to restore pre-Katrina levels of hurricane protection to New Orleans by the  June 1 start of the 2006 Atlantic hurricane season (which generally runs  June 1 start of the 2006 Atlantic hurricane season (which generally runs  from June 1 to November 30 each year), the U.S. Army Corps of Engineers  from June 1 to November 30 each year), the U.S. Army Corps of Engineers  (Corps), in late 2005, considered strengthening the drainage canal  (Corps), in late 2005, considered strengthening the drainage canal  floodwalls but decided against this effort due to cost and time constraints.  floodwalls but decided against this effort due to cost and time constraints.  Instead, the Corps decided to install three interim closure structures or  Instead, the Corps decided to install three interim closure structures or  gates near the points where the canals meet the lake. These gates would  gates near the points where the canals meet the lake. These gates would  be closed during major hurricane events to prevent storm surge from  be closed during major hurricane events to prevent storm surge from  entering the canals and potentially overtopping or breaching the canal  entering the canals and potentially overtopping or breaching the canal  floodwalls and flooding the city. When the gates are closed, however,  floodwalls and flooding the city. When the gates are closed, however,  rainwater cannot drain from the three canals into Lake Pontchartrain, and  rainwater cannot drain from the three canals into Lake Pontchartrain, and  large-capacity pumping systems are needed to pump water out of the  canals and into the lake. Due to space constraints along the canals and the  limited amount of time it had before the start of the 2006 hurricane season,  the Corps decided to procure 34 large-capacity hydraulically powered  pumping systems to provide the most pumping capacity possible by June  1, 2006. In mid-2006, the Corps modified the contract to procure six  additional hydraulic pumping systems, bringing the total number of  hydraulic pumping systems to be installed along the three drainage canals  to 40.
During the process of acquiring, testing, and installing the hydraulic  pumping systems for the drainage canals, issues with the operation of  these pumping systems came to light. In response to your request, we  issued a report on May 23, 2007, on the procurement process and award of  the pumping system contract and the status of the efforts to address issues  related to the performance of the pumping systems. In June 2007, the  Corps’ Mississippi Valley Division (MVD) released a technical review  report, known as the Independent Team Report (ITR). This report  identified testing and performance-related issues regarding the hydraulic  pumping systems similar to the issues we had identified in our May report  and, in addition, raised potential issues related to the administration and  documentation of contract modifications. The ITR was conducted by a  three-person technical review team, which consisted of two Corps  engineers from outside MVD and one engineer from MVD who acted as the  team coordinator. According to ITR members, the New Orleans District  was not offered the opportunity to provide official comments on the draft  ITR; rather, an advisory panel from MVD reviewed the draft before its  issuance in June 2007.
In this context, you asked us to update our May 2007 report and consider  the ITR findings in our analysis. Specifically, you asked us to evaluate the  Corps’ efforts to (1) develop the specifications for the pumping systems,  (2) award the contract, (3) address pumping system performance issues  identified during factory and on-site testing, (4) document modifications to  the contract, and (5) reconcile payments made and amounts still owed to  the contractor.
To address these objectives, we reviewed the Corps’ plans for the interim  gates and temporary pumping systems for the three New Orleans drainage  canals. (Unless otherwise noted, the pumping systems discussed in this  report are only the 40 60-inch hydraulic pumps installed at the three  canals.) We reviewed documentation, including e-mails, correspondence,  and other documents related to the solicitation process, contract  specifications and other provisions, factory and on-site test results,  performance requirements, contract modification files, payments to the  contractor, and the Corps’ plans for increasing pumping capacity through  2007. We visited the 17th Street, London Avenue, and Orleans Avenue  Canals and observed the operation of the pumping systems. We  interviewed officials from (1) Corps Headquarters, New Orleans District,  other Corps districts, and members of the MVD’s technical review team  related to the contract and pump performance; (2) Moving Water  Industries (MWI) Corporation and two other pump suppliers that bid on  the solicitation; and (3) the architectural and engineering consulting firms  under contract with the Corps that researched available pumping system  alternatives, qualified pump manufacturers, pump delivery timelines, and  costs and that helped design the canal gates and pumping stations. We  conducted our work from September through December 2007 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Since its founding in 1718, the city of New Orleans and its surrounding  areas have been subject to numerous floods from the Mississippi River and  hurricanes. The greater New Orleans metropolitan area, composed of  Orleans, Jefferson, St. Charles, St. Bernard, and St. Tammany parishes, sits  in the tidal lowlands of Lake Pontchartrain and is bordered generally on its  southern side by the Mississippi River and the Gulf of Mexico. Lake  Pontchartrain is a tidal basin about 640 square miles in area that connects  with the Gulf of Mexico through Lake Borgne and the Mississippi Sound.
Many hurricanes have struck the area over the years, including Hurricane  Betsy in 1965, Hurricane Camille in 1969, Hurricane Lili in 2002, and  Hurricane Katrina in 2005. The hurricane surge that can inundate coastal  lowlands is the most destructive characteristic of hurricanes and accounts  for most of the lives lost from hurricanes. Because of such threats, a series  of flood control structures, including concrete floodwalls and levees, have  been constructed in and around the New Orleans metropolitan area (see  fig. 1).
On August 29, 2005, Hurricane Katrina came ashore near Buras, Louisiana,  about 60 miles southeast of New Orleans, with wind speeds of up to 127  miles per hour and a storm driven wave surge of up to 30 feet. The size and  strength of the storm and subsequent flooding resulted in one of the  largest natural disasters in U.S. history. Storm waters overtopped  floodwalls and levees in Louisiana’s Orleans and neighboring parishes,  causing widespread flooding, many billions of dollars of property damage,  and more than 1,300 deaths. The Corps estimates that more than one-half  of the 269 miles of federally constructed levees and floodwalls in these  parishes were damaged by the storm’s winds and floodwaters.
Through a combination of permanent and temporary measures, the Corps  planned to restore the level of hurricane protection to the New Orleans  area that existed prior to Hurricane Katrina by June 1, 2006. To restore the  pre-Katrina level of protection in a period of about 9 months, the Corps  had to work quickly and, in some instances, engineer temporary solutions  because not all of the repairs could be completed in time. One such  temporary solution was needed along the Orleans East Bank, located  south of Lake Pontchartrain, from the 17th Street Canal to the Inner  Harbor Navigation Canal, and along the western bank of the Inner Harbor  Navigation Canal to the Mississippi River. About 19 miles of levees and  floodwalls are located along the Orleans Lakefront, the Inner Harbor  Navigation Canal, and three drainage canals—17th Street, London Avenue,  and Orleans Avenue—which drain rainwater from New Orleans into Lake  Pontchartrain. A total of about 1 mile of levees and floodwalls was  damaged along the 17th Street Canal and two sides of the London Avenue  Canal, resulting in flooding of New Orleans (see fig. 2).
The city’s three drainage canals are critical to avoid flooding in New  Orleans from a rain storm. During rain events, the city’s Sewerage and  Water Board pumps rainwater from the city into three drainage canals at  17th Street, London Avenue, and Orleans Avenue, which then flows  unrestricted into Lake Pontchartrain. According to the Corps, the  maximum amount of water that the Sewerage and Water Board can pump  into these drainage canals is 10,500 cubic feet per second (cfs) at the 17th  Street Canal, 7,980 cfs at the London Avenue Canal, and 2,690 cfs at the  Orleans Avenue Canal. Because permanent structures and repairs could  not be completed on the three drainage canals by June 1, 2006, the Corps  decided to install temporary pumping systems to provide protection to the  area for 3 to 5 years until permanent structures can be constructed (see  fig. 3). The Corps chose to install three gates and temporary pumping  systems near the points where the 17th Street, London Avenue, and  Orleans Avenue drainage canals meet Lake Pontchartrain. These gates are  intended to stop hurricane-induced storm surge from Lake Pontchartrain  from entering the canals and possibly overtopping or breaching the canal  floodwalls, which would flood the city. However, because the gates  prevent the drainage canals from draining water from the city into the lake  when the gates are closed during a hurricane event, temporary pumping  systems are needed to pump water out of the canals and into the lake.
Due to the hurricane damage sustained by the floodwalls bordering the  canals, the Corps established the following safe water levels for each of  the drainage canals—6 feet for the 17th Street Canal, 5 feet for the London  Avenue Canal, and 8 feet for the Orleans Avenue Canal. The water level in  each of these canals must be maintained at or below the safe water level in  order to ensure that the already weakened canal floodwalls are not  breached. Further, the total capacity of the temporary pumping systems at  the interim gated closure structures that is necessary to accommodate a  10-year rainfall event without exceeding the safe water levels is 7,700 cfs  at the 17th Street Canal, 5,000 cfs at the London Avenue Canal, and 1,900  cfs at the Orleans Avenue Canal. The hydraulic pumping systems installed  by the Corps at the Orleans Avenue Canal were sufficient to maintain the  safe water levels during a 10-year rainfall event. However, the hydraulic  pumping systems installed at the 17th Street and London Avenue drainage  canals could provide about 4,000 cfs and 2,700 cfs, respectively. In order to  ensure that each pumping station had the needed capacity to pump  enough water during a 10-year rainfall event, the Corps used a separate  contract to acquire and install an additional 11 direct drive pumps and 14  portable hydraulic pumps at the 17th Street Canal, increasing the capacity  from about 4,000 cfs to about 9,200 cfs. The Corps also installed 8  additional direct drive pumps at the London Avenue Canal, increasing the  capacity from about 2,700 cfs to about 5,200 cfs. Table 1 provides the total  number of pumps and pumping capacity at the 17th Street, London  Avenue, and Orleans Avenue Outfall Canals.
Although these additional pumps allow the total pumping capacity at the  three canals to maintain the capacity needed to pump water out of the  canals during a 10-year rainfall event, the capacity is still not sufficient to  match the maximum pumping capacity of the Sewerage and Water Board’s  pumps. As a result, during a hurricane event, some flooding might occur in  some parts of the city from rainfall, although it is likely that this flooding  would be significantly less than that which occurred from the overtopping  and breaches of the canal walls during Hurricane Katrina. Appendix II  provides the pumping capacity trends for the 17th Street, London Avenue,  and Orleans Avenue drainage canals from June 1, 2006, through November  30, 2007.
section_title: Commitment to Meet Schedule Drove Specifications but Resulted in Deficiencies in Some Key Contract Terms
paragraphs: The Corps’ efforts to develop the specifications for the pumping systems  were driven by its commitment to have as much pumping capacity as  possible in place at the drainage canals by June 1, 2006—the start of the  first Atlantic hurricane season after Hurricane Katrina. Due to the  compressed schedule and the limited space available for installation, and  based on the limited market research conducted by the Corps’ consultants,  the Corps decided to use 60-inch hydraulic pumping systems rather than  alternatives that would have involved longer delivery schedules or  required more space. The Corps’ consultants drafted contract  specifications that closely matched those of one supplier, which, along  with the 60-inch pumping system requirement, resulted in that supplier  being in the strongest position to compete for the contract. Further, the  contract itself was not written as precisely as it should have been.  Specifically, the original factory test requirements were ambiguous, there  were limited provisions for on-site testing, and there were no criteria for  acceptance of the pumping systems by the government.
subsections: section_title: Space and Schedule Requirements Were Key Considerations in Developing Specifications
paragraphs: The decisions made by the Corps during the procurement of pumping  systems at three New Orleans drainage canals were driven largely by  space and schedule considerations. The Corps began the acquisition  process by contracting with two architectural and engineering consultant  firms (consultants) to determine available technical options that could  meet the Corps’ schedule, space, and pumping capacity needs; conduct the  associated market research; and survey pump equipment suppliers. On the  basis of their technical analysis, the consultants concluded that the use of  hydraulic-driven pumps was the best alternative for the Corps because  electric-driven direct drive pumps would need auxiliary equipment that  would require more space for installation and would have a longer delivery  time. They also determined that using hydraulic pumps less than 60 inches  in diameter would require more pumps to be installed and require added  space to provide the same amount of pumping capacity.
section_title: Specifications Closely Matched Those of One Supplier, but Key Contract Provisions Were Deficient
paragraphs: Corps consultants drafted contract specifications that closely matched  those of one supplier. The consultants conducted limited market research  and found that at least two suppliers had specifications for a 60-inch  hydraulic pump. One of those suppliers was MWI, a company that the  consultants had spoken with as they were developing the design for the  gates and pump stations along the drainage canals. The consultants met  with MWI and also contacted at least two other pump manufacturers  regarding their pumps. Of these suppliers, the consultants identified MWI  as the only supplier who had actually manufactured a 60-inch hydraulic  pump with a 60-inch impeller, the mechanism that drives water through  the system. Another pump manufacturer had a design for a 60-inch pump,  but it included only a 54-inch impeller. The consultants believed that MWI  could deliver the 34 60-inch pumping systems that the Corps needed on  schedule.
The Corps did not have an existing technical specification for a 60-inch  hydraulic pump. The consultants drafted a specification for the Request  for Proposals (RFP) based on technical specifications and descriptions of  the pumps contained in catalogs published by MWI and another  manufacturer. The consultants told us that they had provided the Corps  with a generic specification because any reference to a specific supplier  had been removed. However, our analysis of the RFP’s equipment  specifications indicates that they more closely matched MWI’s than the  other manufacturer’s catalog descriptions. In fact, the testing  specifications used for the RFP were nearly identical to those published by  MWI, which included an open sump test requirement. After the other  manufacturer complained that the open sump test requirement was  restrictive because only MWI had an open sump, the Corps amended the  RFP to delete this requirement. This open sump test requirement was  incorporated into the contract at the time of award, however, because it  was offered by MWI as part of its proposal.
Other contractual testing and acceptance criteria were ambiguous,  inadequate, or missing altogether. Specifically, the contract did not clearly  state whether factory flow and head testing was required of each pump,  the on-site testing requirement merely stated that there should be no leaks,  and there were no final acceptance criteria in the contract. Terms and  conditions in contracts should be clear and complete so that the parties  fully understand their obligations and that potential disputes can be  avoided. To date, Corps and MWI have been able to address identified  deficiencies in the contract, which were largely caused by the perceived  need to move forward expeditiously. However, the extent to which these  or other contract issues may lead to disputes between the parties will not  be known until the time of contract closeout, currently scheduled for early  2008.
section_title: The Corps Used a Streamlined Solicitation Process and Awarded the Pumping Systems Contract to the Highest-Rated Competitor
paragraphs: Given the need to procure and install the temporary pumping systems  before the June 1 start of the 2006 hurricane season, the Corps decided to  use a streamlined process to contract for the pumping systems. Like most  other federal agencies, the Corps has statutory authority to use other than  full and open competition procedures when the agency’s needs are of an  unusual and compelling urgency. Using this authority, the Corps  streamlined parts of the acquisition process. The RFP was issued on  January 13, 2006, and required that the contractors’ proposals be  submitted by January 18, just 5 days later. Normally the solicitation would  allow for a response period of at least 30 days.
The Corps received three proposals in response to its RFP. Suppliers  submitted pricing information and technical proposals and made oral  presentations to the Corps. The Source Selection Evaluation Board,  whose voting members consisted of three Corps officials, evaluated offers  using four technical evaluation factors identified in the RFP in descending  order of importance: (1) technical approach, (2) project management, (3)  past performance, and (4) small business or small disadvantaged business  participation. The solicitation also provided that, when combined, these  technical evaluation factors were weighted approximately equal to price.
The Source Selection Evaluation Board rated MWI’s proposal significantly  higher than the other two proposals. MWI’s proposal included  commitments from suppliers and subcontractors to deliver the pump  components needed by MWI to assemble the pumps. The Corps believed  MWI represented the best chance of meeting the Corps’ critical deadline of  June 1, 2006. MWI offered a price of $26.6 million, which was within 2.8  percent of the government estimate of $25.6 million. The contracting  officer determined that MWI’s price was fair and reasonable and awarded  a firm, fixed-price contract to MWI on January 27, 2006. The contract also  contained an incentive of up to $5 million that MWI could earn for early  delivery. To date, the Corps has increased the contract price by about $6  million for required pump modifications and for six additional pumping  systems, bringing the total number of hydraulic pumping systems acquired  to 40. Figure 4 shows a diagram of the hydraulic pumping system.
section_title: The Corps and the Contractors Have Addressed Pumping System Testing and Performance Issues
paragraphs: The Corps and its contractors have addressed and corrected the pumping  system testing and performance issues identified by both our May 2007  report and the ITR. Factory testing, which occurred from March 2006  through May 2006, revealed several issues with some components of the  pumping systems, and concerns were raised that the pumping systems  would not perform as intended. On May 2, 2006, the Corps modified the  original contract, replacing the original testing requirements with new  procedures because of schedule and performance concerns. Beginning in  June 2006, however, even though all of the problems identified during  factory testing had not been resolved, the systems were installed as  planned because the Corps believed it was better to have some pumping  capacity along the drainage canals during the 2006 hurricane season rather  than none. The Corps also thought that most of the issues identified during  factory testing could be resolved after installation. The Corps and the  contractors took several steps to correct the known performance issues  after installation and, as of September 2007, all of the pumping systems  have been reinstalled and all of the outstanding repairs have been  completed. According to Corps officials, the results of on-site testing now  show that the 40 hydraulic pumping systems are fully operational and final  acceptance of the pumping systems is scheduled for the beginning of  calendar year 2008.
subsections: section_title: Factory Testing Requirements Were Modified to Focus on Deficiencies Identified
paragraphs: On May 2, 2006, the Corps issued modification No. 4, “revised test  procedures,” to the contract. According to the contracting officer  responsible for oversight of the pumping system contract, these revised  testing procedures replaced the original factory testing requirements with  new testing requirements. The contract initially required each pump and  hydraulic power transmission system to be factory pressure tested  statically and dynamically. In addition, full-size flow and head testing was  to be witnessed by the government prior to shipment of the pumping  systems. The pump flow and head testing was to be conducted in an open  sump at the manufacturer’s testing facility in accordance with Hydraulic  Institute (HI) standards and in the presence of a registered professional  engineer. According to the contracting officer, modification No. 4 replaced  these testing procedures. The modification required, among other things,  testing the hydraulic drive units for a minimum of 3 hours and utilizing  previous model tests of the pump design to predict the pump capacity.  Further, the modification required all pumps to be pressure tested for 90  minutes.
According to New Orleans District Corps officials responsible for  oversight of the contract, the original testing requirements were  interpreted by a Corps inspector and the ITR to include full-size flow and  head testing of each of the pumps. Furthermore, the ITR concluded that  modification No. 4 did not specifically delete the original testing  requirements and, therefore, assumed the testing that had been conducted  did not meet the contract requirements regarding full-size flow and head  testing for each pump in accordance with HI standards. Based on this  assumption, the ITR concluded that the contractor owed the government a  refund because it had not completed the testing required in the contract.  The ITR’s reading of the modification may have overlooked the  modification’s purpose, however, which was to adjust the required testing  to focus on those elements of the pumps in need of further refinement,  given the limited time available. We believe the ITR may have reached this  conclusion because it did not discuss the intent of the “revised test  procedures” modification to the original testing requirements with either  the contracting officer or the Corps’ technical officials. Officials from the  New Orleans District told us that it was never the intention of the Corps to  interpret the testing requirements as requiring every pump to be full-size  flow and head tested in accordance with HI standards–only static and  dynamic tests were originally required of each pump. Corps technical and  contracting officials said the revised testing procedures contained in the  modification were developed to focus on the mechanical issues that had  been identified and, therefore, replaced the original testing requirements,  which included allowing the use of model test results in lieu of HI tests.
section_title: Pumping System Performance Issues Have Been Addressed by the Corps and the Contractors
paragraphs: The Corps and the contractors have addressed and corrected the concerns  raised about some components of the pumping systems during factory and  on-site testing. As noted in our May 2007 report and in the ITR, the primary  concerns identified during testing included undersized gear oil circulation  motors, hydraulic motor vibrations, the design of the hydraulic intake line,  suspect pipe welds, and lower than expected pumping capacity. As a result  of the concerns identified during testing, the Corps had no assurance that  the pumping systems would operate to capacity if needed during the 2006  hurricane season. Nevertheless, the pumping systems were installed as  planned because the Corps believed it was better to have some pumping  capacity along the drainage canals during the 2006 hurricane season rather  than none. On June 1, 2006, the Corps had installed 11 pumping systems,  and by July 2006, it had installed 34, although it is uncertain how much of  the theoretical capacity of these pumping systems would have been  available, and for how long, if needed during the 2006 hurricane season.  The Corps also believed that many of the issues identified during factory  testing could be resolved after installation. After installation, the Corps  and its contractors took several steps to correct known performance  issues with the pumping systems. The main performance issues, and the  ways the Corps and the contractors addressed each of them, are described  in more detail below.
During factory testing, the Corps observed that the gear oil circulation  pump motors were overheating, which resulted in the failure of some of  the motors. MWI determined that the pump motors were too small. All of  the motors were eventually replaced with larger gear oil circulation  motors, resolving the problem.
During on-site testing in August 2006, the hydraulic motors were  experiencing greater than normal vibrations. According to the ITR, this  condition could have led to the failure of the equipment. Initial analysis of  the problem indicated that there may have been a hydraulic short in the  Rineer hydraulic motor that drives the main pump impellers. The motor  manufacturer made modifications to the motor, and preliminary testing of  the motors in late August 2006 appeared to confirm that these  modifications eliminated the vibrations. However, upon further testing,  vibrations were still present to varying degrees. Additional on-site testing  was performed, and in late November 2006, it was determined that the  vibrations were due to undersized springs in the Rineer hydraulic motors.  The motor manufacturer replaced the undersized springs with heavier  springs. According to Corps officials, on-site tests witnessed by the  government after the installation of the new springs and measurements  conducted by a third-party contractor document that the pumping systems  now operate with no apparent vibration issues.
Because of concerns that the hydraulic intake lines could adversely affect  pumping performance, the Corps requested that MWI redesign and  reinstall the hydraulic intake lines on all of the pumping systems. During  factory testing, the Corps observed a high rate of failure of the Denison  hydraulic pumps on the drive units. The Denison motors pump the  hydraulic fluid from a reservoir to the Rineer motor which then turns the  pump impeller. A preliminary assessment revealed that the majority of the  issues identified in the factory were caused by air entrainment (or dry run  condition) in the hydraulic pumps. The dry run condition was attributed to  air getting into the hydraulic system upon initial start-up of the drive unit.  To eliminate the dry run issue, two interim changes were made to the  system until a more permanent fix could be implemented: (1) a check  valve was installed on all of the hydraulic intake lines, and (2) the pump  start-up procedure was modified so that the system was started at a lower  speed and gradually increased to the normal operating speed.
The ITR concluded that the pumping systems would probably not have  performed as designed because the inclusion of a check valve would  require priming the pump prior to start-up and the original intent of the  design was to allow for unmanned operation of the equipment. Both Corps  and MWI officials stated that the ITR was incorrect in assuming that the  pumps would have to be primed using the check valve at every start-up.
Instead, these officials stated that the pumping systems would have  operated as intended because using this valve to prime the Denison pumps  is only necessary immediately after maintenance is performed on the  system. Additionally, according to a Lake Borgne Levee District official,  this pump design has been successfully used for about 20 years without  having to prime the pumps prior to start-up. Nevertheless, in order to  ensure that air would not be pulled into the hydraulic pumps, causing  failure of the system, the Corps requested that MWI redesign the hydraulic  intake system to provide for a flooded suction without a siphon. Figure 5  shows a drawing of the original hydraulic pump design with siphon.
Corps officials from the New Orleans District emphasized to us that the  redesign was requested to more adequately meet their needs, not because  of concerns about the pumping systems operating as intended. MWI  subsequently agreed to modify the design of the hydraulic intake line at  the request of the Corps. According to Corps officials, by the end of July  2007 and at its own expense, MWI had redesigned and reinstalled the new  flooded suction design on all 40 pumping systems (see fig. 6).
Because of questionable welds identified on the pump housing, the Corps  decided to replace certain welds to ensure they would not fail during  pump operations. Upon inspection of the pump housing, the Corps  determined that some of the welds on the pump housing may not be  sufficient. While MWI provided the Corps with a “fit for service letter” for  all of the welds on the pump housing and an extended warranty, the Corps  decided that it was prudent to replace the welds on the pump housing  below the base plate (the segment of the pump that is below the water  level) in order to ensure that the welds would not fail during pumping  operations. All of the necessary welds have been corrected, and the Corps  plans to negotiate this additional cost during contract closeout.
Additionally, issues were raised about the adequacy of the welds on the  hydraulic piping, which carries high pressure hydraulic fluid from the  Denison pump to the Rineer motor. The hydraulic piping was subsequently  visually inspected and pressure tested to 1.5 times its operating pressure  as part of the quality control process. The testing results indicated that the  piping was adequate for transmitting power from the diesel engine to the  water pump.
Initial pumping capacity testing indicated that the pumping systems may  not have been performing at the design capacity level. In April 2006, MWI  conducted full-size factory flow and head tests on the hydraulic pumps. A  representative from the Corps’ Engineering Research and Development  Center (ERDC) reviewed these test results and concluded that the test  results showed that the pumps would operate at about 96 percent of the  specified capacity. However, according to the ITR, these tests were not  conducted in accordance with HI standards and, therefore, were invalid.
In August 2006, on-site flow and head testing was conducted at the canals.  In order to test the pumping systems, the interim gates were closed and  water was pumped into the canal by the city’s Sewerage and Water Board  to raise the water level in the canal to the elevation necessary for the  pumping systems to be tested. However, because adequate water levels in  the canal could not be achieved to replicate design conditions, the pumps  could not reach a fully primed condition. The Corps decided to invert the  discharge pipes in order to enable the pumps to reach a primed condition  with less than design water conditions in the lake. This facilitated testing  of the pumping systems and allowed measurements to be recorded and  analyzed.
In September 2006, a representative from ERDC was consulted and  performed on-site flow and head tests of pumping systems at the London  Avenue canal. A month later the Corps and ERDC performed the same  tests at the 17th Street canal. Data collected from these on-site tests  revealed that the pumping systems were working near the appropriate  capacity. Based upon the on-site testing results and upon suggestion from  the ERDC representative, all of the discharge pipes at all of the canals  were inverted and cut at a 30 degree angle, which allows the pumps to  prime at lower canal water elevations and enhances the flow rates (see fig.  7).
In November 2006, another full-size factory flow and head test was  conducted by MWI and ERDC. However, due to constraints at the testing  facility, the full-size factory test, which was done in consultation with the  ERDC representative, was completed with deviations from the HI  standards. This test revealed that the pumping capacity ranged from 93.6  to 97.6 percent of the design specification and performed without  problems during the 2 days of testing. According to Corps officials, MWI  further agreed to construct a model test to confirm the pumping systems  would perform within HI standards. In September 2007, a Corps ERDC  official witnessed a model test conducted by MWI and prepared a report,  which concluded that the pumping systems would operate at 98.6 percent  of the design capacity. According to Corps officials, these results are  within acceptable limits and any issues remaining with the final pumping  capacity will be negotiated at contract closeout.
section_title: Pumping System Acceptance Is Near Completion
paragraphs: According to Corps officials, the Corps plans to make final acceptance of  the pumping systems during the beginning of calendar year 2008. The  original pumping system contract lacked clearly defined on-site testing  procedures, requiring only that the pumps and hydraulic equipment be  tested for leaks. In light of the various issues surrounding the pumping  systems, the Corps and MWI agreed that it was necessary to show that all  of the pumping systems could operate at a steady state after installation.  According to Corps and MWI officials, a major challenge with on-site  testing of the pumping systems is simulating the amount of water that  would be present in the canals and the lake during a storm event. Under  normal conditions, when there are low water levels in the canals, it is not  possible to test each pump system for an extended period of time, and any  tests conducted cannot approach the design capacity of the pumping  systems. Due to this limitation, the Corps subsequently developed specific  pumping system acceptance testing procedures that, among other things,  include running each pumping system continuously for 2 hours. Corps  officials told us that because most of the issues associated with the  pumping systems occurred within the first 45 minutes of operation, the 2- hour testing period for each pumping system was sufficient.
In its June 2007 report, the ITR team concluded that at the time of their  review in September 2006, the pumping systems would not perform as  intended because of issues encountered in factory testing in early 2006.  Since September 2006, there have been a number of analyses, changes,  and additional testing of the pumping systems to address these earlier  concerns. For example, between November 2006 and September 2007, the  Corps had completed all of the repairs that were outstanding at the end of  the 2006 hurricane season and which were noted in the ITR, and  reinstalled all 40 pumping systems. In addition, as of September 2007, each  pumping system had been successfully tested on site for at least 2 hours,  providing greater assurance that they will perform as designed during  future hurricane seasons. On September 27, 2007, GAO officials witnessed  the pumping systems performing at both the 17th Street and London  Avenue Canals (see fig. 8). According to Corps officials, because all of the  outstanding repairs have been completed and on-site testing indicates that  the system is now fully operational, final acceptance of the pumping  systems and the contract closeout is expected to be completed early in  calendar year 2008.
section_title: Contract Files Remained Incomplete for Months, but Currently Contain Required Documentation for the Type and Value of Procurement
paragraphs: Contract files for the pumping systems, although incomplete at the time of  the ITR review, now contain the required documentation for the type of  contract and the value of the associated modifications. In a number of  cases, however, Corps officials inserted required documentation in the  contract files several months after modifications were issued and only  after the ITR reported its findings. While the ITR correctly noted the  absence of some forms of required documentation, we found that much of  the documentation specifically cited—including requests for proposals,  independent government estimates, certified cost or pricing data, technical  analyses, and price negotiation memorandums—was not required for the  modifications in question. In addition, while the ITR found that it appeared  as though the contractor developed the scope of work and pricing for  some of the modifications without a subsequent analysis by the Corps, we  found no instance of this occurring. Rather, our review found that, for  most of the contract modifications there was extensive back and forth  discussion, usually by e-mail, between officials from the Corps and MWI.
subsections: section_title: Significant Documentation Added to the Contract Files Only after the ITR Review
paragraphs: The ITR team reviewed 18 of the first 30 contract modification files and  reported that many lacked significant documentation. Specifically, the ITR  identified 13 modification files with deficiencies—most pertaining to  documentation of the Corps’ determination of fair and reasonable pricing.  Our review confirmed that significant documentation was added to the  files only after the ITR team issued its report. We reviewed the files for the  32 post-award modifications, focusing in depth on the files related to the  13 modifications found by the ITR team to contain deficiencies, as well as  2 additional modifications that were issued after our May 2007 report and  the ITR review. Of the modifications we reviewed in depth, 10 contained  internal memorandums, prepared by the contracting officer after the fact,  to document price reasonableness or the events supporting the  modification. Another 2 modifications contained undated memorandums  of price reasonableness signed by the contracting officer. Finally, of the  eight purchase request and commitment forms on file, five were prepared  on the same date to retroactively document the availability of funds for  modifications that were issued 9 to 17 months earlier. Documentation in  some of the files, however, suggests that the availability of funds was  determined through other means at the time the modifications were  signed.
In response to the ITR, the Corps’ contracting officer acknowledged that  the contract files could have been better managed but stated the Corps felt  it was more important to get the pumps installed in a timely manner. In  order to do this, the Corps issued the modifications with the intention of  settling all outstanding issues with the contractor before closing out the  contract. The Corps agreed with the ITR, however, that certain  documentation was missing and took corrective actions to complete the  files. The contracting officer, whom the ITR team did not meet with for  their review, noted that because many of the people working on the  pumping systems procurement were rotating through the District Office,  they may not have completed or submitted all of the necessary paperwork  before leaving. Even though it is currently complete, preparing  documentation months after an event occurs increases the likelihood that  the documentation may contain inaccuracies or ambiguities, which make  it difficult to resolve any disputes that may arise.
section_title: Many Specific Documents Cited by the ITR as Missing Were Not Required
paragraphs: As of October 2007, the contract modification files appeared up to date  and consistent with Federal Acquisition Regulation (FAR) requirements.  While the ITR correctly noted the absence of some forms of required  documentation, much of the documentation specifically cited by the ITR— including requests for proposals, independent government estimates,  certified cost or pricing data, technical analyses, and price negotiation  memorandums—was not required for the modifications in question. In  some respects, it appears the ITR treated the pumping systems contract as  if it were for construction rather than supplies. Different documentation  requirements apply to these types of contracts.
Ten of the modifications we reviewed in-depth increased contract costs  and, therefore, required documentation of fair and reasonable pricing.  While independent government estimates are one technique that can be  used to analyze price and are required for construction contracts, they are  not specifically required for supply contracts, such as the contract for the  pumping systems. Nonetheless, the Corps obtained—and included in the  files after the ITR review—independent government estimates for six of  the modifications.
None of the 10 modifications with additional costs that we reviewed in- depth required the contractor to provide certified cost or pricing data.  Specifically, we found that 7 of the modifications fell under the threshold  requiring cost or pricing data. The contracting officer determined that  cost or pricing data was not required for another modification because it  combined separately priced changes from 2 previous modifications that  were each below the threshold. Finally, for 2 modifications related to the  purchase of six additional pumps, the contracting officer concluded that  adequate price competition existed from the base contract and, therefore,  additional pricing data was not required. At least some information on  pricing provided by the contractor was included in the files for all 10 of the  modifications that involved additional costs.
According to the FAR, when contractor certified cost and pricing data are  not required, price analysis shall be used to determine a fair and  reasonable price. While the FAR provides numerous analysis techniques,  including the use of independent government estimates, it does not require  the use of any one method. For 8 of the modifications we reviewed, the  Corps’ contracting officer documented price analysis and negotiations  with the contractor through signed internal memorandums for the files,  and for 2 modifications, used price negotiation memorandums. In  addition, while not required, the Corps obtained internal technical  analyses for 3 of the modifications we reviewed in depth to determine the  reasonableness of MWI’s proposals. Table 2 summarizes GAO’s analysis of  the ITR’s findings regarding missing documentation in the contract files.
section_title: File Documentation Establishes Independence of Corps Decision Making
paragraphs: In addition to contract documentation issues, the ITR also reported that it  appeared, in some circumstances, as though the contractor developed the  scope of work and pricing for the modifications without a subsequent  analysis by the Corps. We found no instance of this occurring. Rather, our  review of the files indicate that, for most of the contract modifications,  there was extensive back and forth discussion, usually by e-mail, between  personnel from the Corps and MWI. These discussions focused on the  causes of and solutions to technical issues, as well as the costs of  corrective actions.
While each of the modifications was unique, modification No. 2 is  illustrative of many of the contract modifications we reviewed.  Specifically, shortly after award of the contract, the Corps determined that  it needed the capability to control the pumps from a remote location, since  in the event of a hurricane the operator would be required to seek shelter  in a control booth. The Chief of Engineering from the Corps prepared a  request to modify the contract to require master pump control panels. The  request contained detailed specifications of what was required and  estimated that the additional cost would be $150,000. The contracting  officer sent the request to MWI and asked for a cost proposal. MWI replied  through an e-mail that repeated the specifications provided to it by the  Corps and offered a price of $188,699. The Corps requested additional  support for the price, and MWI responded with a copy of the quote it had  received from its supplier, pricing for MWI’s markup, and the additional  work MWI would perform. A Corps engineer reviewed this information  and informed the contracting office that MWI’s proposed price was  reasonable. The Corps issued a contract modification with the  specifications it developed at the price quoted by MWI. As was the case for  a number of modifications, there was no contemporaneous price  reasonableness document signed by the contracting officer; rather, an  undated “after the fact” memorandum concluding that MWI’s price for the  modification was reasonable was added to the file.
section_title: The Corps Has Not Overpaid the Contract and Has Plans to Reconcile Payments Made in Error
paragraphs: As of October 31, 2007, the Corps had paid the contractor about $30.5  million of the $33 million contract for the 40 hydraulic pumping systems  and has plans for reconciling mistaken payments it made. The Corps made  payments to the contractor only after receiving invoices from the  contractor for delivered items and services. In most cases, the Corps only  paid 80 percent of each invoice and held the other 20 percent as retained  funds in order to ensure the contractor was not overpaid and that any  performance issues were addressed.
The ITR identified a few instances where the contractor had received  payment more than once for the same item. Our review confirmed that this  did occur. We found, however, that these payments were made in error by  the Corps and did not indicate any improper behavior on the contractor’s  part. Specifically, on December 6, 2006, the Corps received one invoice  requesting payment for three drive units and three pumps valued at about  $2.2 million because they were complete, and MWI believed that they  could be delivered if the Corps wanted them at that time. On the same day,  the Corps notified MWI that it could not pay for the pumps and drive units  until they were actually delivered. MWI then e-mailed the Corps,  requesting that they ignore the original invoice and stating that they would  send new invoices for the drive units and pumps upon shipment. The  Corps subsequently received three separate invoices, each requesting  payment for one drive unit and one pump. However, the Corps paid all of  the invoices, including the invoice that the contractor told them to ignore.  As a result, the Corps paid twice for the same three pumps and three drive  units. According to the Corps’ contracting officer, the duplicate payments  will be corrected by deducting the balance from withheld funds and not  paying some outstanding invoices. Our review found no other instances  where duplicate payments were made to the contractor. We also found 14  instances where the contractor sent invoices to the Corps for work  completed, which have not been paid. The net effect is that the contractor  has not been overpaid under the contract.
On June 8, 2007, the Corps sent a letter to MWI providing an explanation  as to why the Corps had not paid these outstanding invoices, and  describing how the Corps planned to reconcile the duplicate payments  made in January and February 2007 by subtracting the amount of the  outstanding invoices from any additional invoices it received. From July  through October 2007, the Corps made four additional payments to the  contractor from the payments it had withheld, totaling about $1.8 million.  The Corps has still not made final payment for the outstanding amount  remaining on the contract. In addition, the Corps has withheld payments  related to an early delivery incentive of approximately $5 million until the  final acceptance of the pumping systems. According to Corps officials,  the final payment and reconciliation of the contract, including any  incentive payments or penalties, will be settled with the contractor after  final acceptance of the pumping systems. The Corps expects this to take  place in the early part of calendar year 2008.
section_title: Conclusion
paragraphs: The Corps’ actions in awarding and administering the pumping system  contract were generally in accordance with federal requirements.  However, in its haste to award the contract and acquire and install the  pumps, the Corps did not develop a contract that was clear and precise  with respect to testing and acceptance criteria and did not always  promptly prepare required contract related documents. In some cases, this  has led to uncertainties about exactly what was required of the contractor  to comply with the contract’s terms and conditions. This also creates the  potential for contract disputes, which can be difficult, expensive, and time- consuming to resolve. In addition, in those cases where required  documents were prepared “after the fact,” there is an increased likelihood  that documents prepared months after events have occurred may contain  inaccuracies as memories have faded and key personnel may have moved  on to other positions.
section_title: Recommendations for Executive Action
paragraphs: While we recognize that this procurement was conducted under exigent  circumstances, we believe that the procedures used by the Corps could be  improved for future procurements. For this reason we recommend that the  Secretary of Defense direct the Commanding General and Chief of  Engineers of the U.S. Army Corps of Engineers to:  take steps, through additional guidance or otherwise, to reinforce the  importance of adherence to sound acquisition practices, even during  expedited procurements, including ensuring that important contract  provisions, such as any required testing, are clear so that the contractor  and the government understand what conditions or criteria must be met  for successful completion of the contract; and   develop procedures to ensure that any required contract-related  documentation, including that related to contract pricing, is completed  and filed within a reasonable period of time.
section_title: Agency Comments and Our Evaluation
paragraphs: The Department of Defense provided written comments on a draft of this  report, which are reprinted in appendix III.  The Department of Defense  concurred with our recommendations and provided information on what  actions it would take to address them.  Concerning our recommendation  to adhere to sound acquisition practices, the Department of Defense said  the Secretary of Defense will direct the Corps to send guidance to all  Corps offices emphasizing the need for clearer technical specifications so  that the contractor and government understand what conditions or criteria  must be met for successful contract completion.  To address our  recommendation to ensure more timely completion of required contract  file documentation, the Department of Defense said the Secretary of  Defense will direct the Corps to review and revise as necessary current  policies and regulations.  The Department of Defense also provided us  with technical comments, which we have incorporated throughout the  report, as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date.  At that time, we will send copies to interested congressional  committees; the Secretary of Defense; and the Commanding General and  Chief of Engineers of the U.S. Army Corps of Engineers. We will also make  copies available to others on request.  In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or any of your staff have any questions about this report, please  contact one of us at (202) 512-3841 or mittala@gao.gov, (202) 512-4841 or  woodsw@gao.gov, or (202) 512-6923 or dornt@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To evaluate efforts by the U.S. Army Corps of Engineers (Corps) to solicit,  award, and administer the pumping system contract, we reviewed the  Corps’ plans for the interim gates and temporary pumping systems  consisting of the 40 hydraulic pumps installed at the three New Orleans  drainage canals. We also reviewed applicable Federal Acquisition  Regulation criteria, especially pertaining to contract pricing; the contract  and specifications; e-mails, correspondence, and other supporting  documentation related to the solicitation and award of the contract;  factory and on-site test results; performance requirements; the 32 contract  modifications and supporting documentation; the Mississippi Valley  Division (MVD) Independent Team Report (ITR); the Corps project  delivery team’s response to the ITR; contractor invoices and payment  records; and the Corps’ plans for increasing pumping capacity through  2007.
We also visited the 17th Street, London Avenue, and Orleans Avenue  Canals and observed the operation of the pumping systems. We  interviewed contracting and program officials from (1) Corps  Headquarters, New Orleans District, other Corps districts, and members of  the MVD’s technical review team related to the contract and pump  performance; (2) Moving Water Industries Corporation and two other  pump suppliers that bid on the solicitation; and (3) the architectural and  engineering consulting firms under contract with the Corps that  researched available pumping system alternatives, including qualified  pump manufactures, pump delivery timelines, and costs, and that helped  design the canal gates and pumping stations. We conducted our work from  September through December 2007 in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Pumping Capacity for the Three Drainage Canals
paragraphs: Appendix II: Pumping Drainage Canals  Cubic feet per second (cfs)
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contacts named above, Ed Zadjura, Assistant Director;  Matthew Reinhart; Katherine Trimble; Christine Frye; James Dishmon;  Rich Johnson; Marie P. Ahearn; and Kenneth E. Patton made significant  contributions to this report.