Coast Guard: Changes to Deepwater Plan Appear Sound, and Program Management Has Improved, but Continued Monitoring Is Warranted
Published: 2006-04-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Deepwater program was designed to produce aircraft and vessels that would function in the Coast Guard's traditional at-sea roles. After the terrorist attacks of September 11, 2001, however, the Coast Guard began taking on additional homeland security missions, and so it revised the Deepwater implementation plan to provide assets that could better meet these new responsibilities. While many acknowledge that the Coast Guard's aging assets need replacement or renovation, concerns exist about the approach the Coast Guard adopted in launching the Deepwater program. The subsequent changes in the program's asset mix and delivery schedules only increased these concerns. This report (1) compares the revised Deepwater implementation plans with the original plan in terms of the assets to be replaced or modified, and the time frames and costs for doing so; (2) assesses the degree to which the operational effectiveness model and other analytical methods used by the Coast Guard to develop the revised Deepwater asset mix are sound and appropriate for such a purpose; and (3) assesses the progress made in implementing GAO's prior recommendations regarding program management. GAO is not making any new recommendations in this report.
section_title: What GAO Found
paragraphs: The revised Deepwater implementation plans change the balance between new and legacy assets, alter the delivery schedule for some assets, lengthen the overall acquisition schedule by 5 years, and increase the projected program cost from $17 billion to $24 billion. The higher cost generally relates to upgrading assets to reflect added homeland security mission requirements. Upgrades to vessels account for the single largest area of increase; with upgrades to the command, control, communications and other capabilities being second highest. In contrast, because the revised plans upgrade rather than replace most legacy aircraft and reduce the number of unmanned aircraft, the cost for Deepwater aircraft drops. The revised plans, like the original plan, are heavily dependent on receiving full funding each year. Coast Guard officials state that a shortfall in funding in any year could substantially increase total costs. The Coast Guard's analytical methods were appropriate for determining if the revised asset mix would provide greater mission performance and whether the mix is appropriate for meeting Deepwater missions. GAO and other independent experts found the Coast Guard's methods were reliable for assessing the effects of changing the asset mix and a Department of Defense review board facilitated accreditation of the Coast Guard's approach. Because the model has proved useful for guiding Coast Guard decisions on the proper asset mix for achieving Deepwater performance goals, the Coast Guard is considering ways to expand the model to guide decisions on meeting its Coast Guard-wide performance goals. Actions by the Coast Guard and the system integrator have fully implemented three of the eight GAO recommendations that were not fully addressed during GAO's review in 2005, and three more recommendations appear to be nearly implemented. The remaining two have unresolved concerns, but the Coast Guard is taking steps to resolve them. A program of this size, however, will likely experience other challenges beyond those that have emerged so far, making continued monitoring by the Coast Guard important.

section_title: Letter
paragraphs: The nation’s new homeland security realities have affected plans for  modernizing the Coast Guard’s fleet of aircraft and vessels. For about a  decade, the Coast Guard has been developing an Integrated Deepwater  System (or Deepwater) acquisition program, a long-term plan to replace or  modernize this fleet. Many of these legacy assets are at or approaching the  end of their estimated service lives. As originally conceived, Deepwater  was designed around producing aircraft and vessels that would function in  the Coast Guard’s traditional at-sea roles, such as interdicting illicit drug  shipments or rescuing mariners from difficulty at sea. After the terrorist  attacks on September 11, 2001, however, these aircraft and vessels began  taking on additional missions related to protection of ports, waterways,  and coastal areas. As a result, the Coast Guard began revising the  Deepwater implementation plan to provide replacement assets that could  better address these added responsibilities. To do so, the Coast Guard  used an operational effectiveness model and other methods to help  determine what mix of assets it needed and what their capabilities should  be. In August 2005, the Coast Guard issued the revised Deepwater  implementation plan detailing the assets it planned to modify or acquire,  along with the proposed cost and schedule for doing so. Then, in February  2006, the Coast Guard again updated its Deepwater plan to align with its  fiscal year 2007 budget submissions.
While there is widespread acknowledgment that many of the Coast  Guard’s aging assets need replacement or renovation, concerns also exist  about the acquisition approach the Coast Guard adopted in launching the  Deepwater program. From the outset, we have expressed concern about  the risks involved with the Coast Guard’s acquisition strategy, which  involves relying on a prime contractor (or system integrator) to identify  the assets needed and then using tiers of subcontractors to design and  build the actual assets. The subsequent changes in the Deepwater asset  mix and delivery schedules only increase these concerns. In 2004 we  reported that well into the contract’s second year, key components needed  to manage the program and oversee the system integrator’s performance  had not been effectively implemented. Accordingly, we made   11 recommendations to address three broad areas of concern: improving  program management, strengthening contractor accountability, and  promoting cost control through greater competition among potential  subcontractors.
This report, prepared at your request, examines the changes the Coast  Guard has made in the Deepwater program to address its broader scope,  as well as addressing the concerns we raised in 2004. More specifically, it    compares the revised Deepwater implementation plan issued in August  2005 and the February 2006 updated plan with the original (August 2002)  plan in terms of the assets to be replaced or modified, and the time frames  and costs for doing so;   assesses the degree to which the operational effectiveness model and  other analytical methods used by the Coast Guard to develop the revised  Deepwater asset mix are sound and appropriate for such a purpose; and   assesses the progress made in implementing our prior recommendations  regarding Deepwater program management.
Our work included extensive reviews and analyses of (1) the original and  revised Deepwater implementation plans, (2) the Coast Guard’s  operational effectiveness models and other analytical tools used for  determining the proposed Deepwater asset mix, and (3) documentation  provided by the Coast Guard on its progress in addressing our  recommendations. We supplemented our document reviews and analyses  with extensive discussions with officials at the Deepwater Program  Executive Office, as well as with interviews of key Coast Guard operations  and maintenance officials, contract monitors, and representatives of the  system integrator. We conducted our work between August 2005 and  March 2006 in accordance with generally accepted governmental auditing  standards. Appendix I describes our objectives, scope, and methodology in  greater detail.
subsections: section_title: Background
paragraphs: As the lead federal agency for maritime homeland security within the  Department of Homeland Security, the Coast Guard is responsible for a  variety of missions, including ensuring security in ports and waterways  and along coastlines, conducting search and rescue missions, interdicting  drug shipments and illegal aliens, enforcing fisheries laws, and responding  to reports of pollution. The Deepwater fleet, which currently consists of  186 aircraft and 88 vessels of various sizes and capabilities, plays a critical  role in all of these missions.
Some Coast Guard Deepwater vessels were built in the 1960s.  Notwithstanding extensive overhauls and other upgrades, a number of the  vessels are nearing the end of their estimated service lives. Similarly, while  a number of the Deepwater legacy aircraft have received upgrades in  engines, operating systems, and sensor equipment since they were  originally built, they too have limitations in their operating capabilities.  The Integrated Deepwater System acquisition program, which the Coast  Guard began developing in 1996, is its major effort to replace or modernize  these aircraft and vessels. This Deepwater program is designed to replace  some assets—such as deteriorating vessels—with new assets, and to  upgrade other assets—such as some types of helicopters—so they can  meet new performance requirements.
The Deepwater program represents a unique approach to a major  acquisition in that the Coast Guard is using a prime contractor—the  system integrator—to identify and deliver the assets needed to meet a set  of mission requirements the Coast Guard has specified. In 2002, the Coast  Guard awarded a contract to Integrated Coast Guard Systems (ICGS), a  joint venture of Lockheed Martin and Northrop Grumman, as the system  integrator for the Deepwater program. Lockheed Martin and Northrop  Grumman, as the two main subcontractors, in turn contract with other  subcontractors. Rather than using the traditional approach of replacing  classes of ships or aircraft through a series of individual acquisitions, the  Coast Guard chose to employ a system-of-systems acquisition strategy that  would replace its deteriorating Deepwater assets with a single, integrated  package of new or modernized assets. This system-of-systems approach is  designed to provide an improved, integrated system of aircraft, vessels,  and unmanned aerial vehicles to be linked effectively through systems that  provide command, control, communications, computer, intelligence,  surveillance, reconnaissance, and supporting logistics. The Deepwater  program’s three overarching goals are to maximize operational  effectiveness, minimize total ownership cost, and satisfy the customer—  the operational commanders, aircraft pilots, cutter crews, maintenance  personnel, and others who will use the assets.
We have been reviewing the Deepwater program for several years,  pointing out successes as well as difficulties and expressing concern over  a number of facets comprising the program. In 2001, we identified several  areas of risk for Deepwater. First, the Coast Guard faced potential risk in  the overall management and day-to-day administration of the contract. At  the time, we reported on the major challenges in developing and  implementing plans for establishing effective human capital practices,  having key management and oversight processes and procedures in place,  and tracking data to measure system integrator performance. In addition,  we expressed concerns about the potential lack of competition during the  program’s later years and the reliance on a single system integrator for  procuring the Deepwater assets. We also reported there was little evidence  that the Coast Guard had analyzed whether the approach carried any  inherent risks for ensuring the best value to the government and if so,  what to do about them.
We reviewed the Deepwater program again in 2004 and found many of the  same concerns. Specifically, we reported that key components needed to  manage the program and oversee the system integrator’s performance had  not been effectively implemented. The Coast Guard’s primary tool for  overseeing the system integrator, integrated product teams (IPT), were  struggling to effectively collaborate and accomplish their missions  because of changing membership, understaffing, insufficient training, and  inadequate communication among members. Also, the Coast Guard had  not adequately addressed the frequent turnover of personnel in the  program and the transition from existing assets to those assets that will be  part of the Deepwater program moving forward. Further, the Coast  Guard’s assessment of the system integrator’s performance in the first year  of the contract lacked rigor, and the factors that formed the basis for the  award fee were unsupported by quantifiable measures. This resulted in the  system integrator receiving an award fee of $4.0 million out of a maximum  of $4.6 million despite documented problems in schedule, performance,  cost controls, and contract administration.
At the time of our 2004 report, the Coast Guard had begun to develop  models to measure the extent to which Deepwater was achieving  operational effectiveness and had reduced total ownership cost, but it had  not made a decision as to which specific models would be used. Further,  Coast Guard officials were not able to project a time frame for when the  Coast Guard would be able to hold the contractor accountable for  progress toward the goals of maximizing operational effectiveness,  minimizing total ownership cost, and increasing customer satisfaction.  Additionally, the Coast Guard had not measured the extent of competition  among suppliers of Deepwater assets or held the system integrator  accountable for taking steps to achieve competition. At the time, the Coast  Guard’s lack of progress on these issues had contributed to our concerns  about the Coast Guard’s ability to rely on competition as a means to  control future programmatic costs. In response to these concerns, we  made a number of recommendations to improve Deepwater management  and oversight of the system integrator. In 2005, we reported that the Coast  Guard had fully addressed three of the recommendations and had actions  underway on others.
For the past several years, the Coast Guard has been revising its  Deepwater plan to incorporate expanded homeland security requirements  it received after the terrorist attacks of September 11, 2001. On May 31,  2005, the Coast Guard submitted a revised implementation plan to the  House Subcommittee on Homeland Security, Committee on  Appropriations, which included both a 20-year and a 25-year plan. The  House Appropriations Committee directed the Department of Homeland  Security and the Coast Guard to select a single revised implementation  plan to accompany the Deepwater fiscal year 2006 budget request. In  compliance with the Committee’s direction, the Coast Guard Commandant  testified on July 21, 2005 to the 25-year revised Deepwater implementation  plan. Further, in February 2006, the Coast Guard submitted an updated  Deepwater implementation plan to align with its fiscal year 2007 budget  submission. These 2005 and 2006 revised plans are the ones we are using  to compare to the Coast Guard’s August 26, 2002, original implementation  plan.
To reflect added homeland security responsibilities based on the terrorist  attacks of September 11, 2001, the August 2005 revision and February 2006  update to the Deepwater implementation plan change the balance of  upgraded legacy versus new assets, the delivery schedules, and program  costs from the original 2002 plan. For aircraft, the revised plans include  upgrading many of the legacy aircraft rather than replacing them with new  assets as called for in the original plan. For vessels, the revised plans  maintain the original plan’s strategy of replacing all of the legacy vessels,  but include some changes in the number of small boats being acquired.  Overall, the revised plan (1) increases the program length by 5 years, to a  total of 25 years; (2) changes the delivery schedules for a number of  assets; and (3) increases overall costs to $24 billion, $7 billion more than  earlier estimates. The program’s higher costs largely reflect the Coast  Guard’s expanded homeland security responsibilities and cover such  changes as greater weaponry, improved communications systems, and  greater operating capabilities. Coast Guard officials caution, however, that  this 25-year program is heavily dependent on receiving the anticipated  budget amount each fiscal year. If full funding is not available in any given  year—for example, because of competing budget priorities—the shortfall  could have cascading effects on overall costs for the Deepwater program.
subsections: section_title: Terrorist Attacks Have Led to Increased Emphasis on Homeland Security and Enhanced Deepwater Asset Capabilities
paragraphs: The original Deepwater plan, while published in 2002, was developed  before the terrorist attacks of September 11, 2001. It reflected an emphasis  on the Coast Guard’s traditional Deepwater missions, such as conducting  search and rescue operations at sea, preventing and mitigating oil spills  and other threats to the marine environment, inspecting foreign vessels,  protecting important fishing grounds, and stemming the flow of illegal  drugs and migrants into the United States. After the events of September  11, 2001, the revised plans took into account the increased security threats  by incorporating a new mission to provide greater security for ports,  waterways, and coastal areas and enhancing the capabilities of the  Deepwater assets to better meet the increased threats. In particular, the  revised plans call for equipping Deepwater helicopters to provide warning  and disabling weapons fire at sea and in ports, waterways, and coastal  areas. Further, while the original plan called for assets to have Deepwater  interoperability—meaning that all Deepwater aircraft and vessels could  communicate with one another—the revised plans call for Deepwater  assets to also have interoperability with assets from the Departments of  Homeland Security and Defense, as well as with the Coast Guard’s  Rescue21 (R21) project. According to Coast Guard officials, this  increased interoperability involves such things as adding circuits and data  transmission capability to allow for more reliable and secure  communication. Table 1 provides further information on some of the key  differences between Deepwater asset capabilities in the original and  revised plans.
section_title: Revised Plans Propose Replacing Fewer Aircraft and Adjusting the Mix of Vessels to Be Acquired
paragraphs: The revised plans change the final mix of Deepwater aircraft more  significantly than the mix of vessels. For example, the original plan called  for replacing all 41 HH-60 Medium-Range Recovery Helicopters with   34 AB-139 helicopters. Under the revised plans, the Coast Guard will  upgrade the HH-60s and not purchase any AB-139 helicopters. Coast Guard  officials said they elected to retain the HH-60s because they determined  that the AB-139 aircraft was unsuitable to meet new requirements for  weaponry and for tactical operations. Retaining and upgrading HH-60  helicopters cost $500 million less than replacing them. Another major  change in aircraft involved retaining more HC-130s to meet long-range  surveillance, search and rescue, and airlift needs. For vessels, the revised  plans retain the original plan’s approach of replacing all cutters and patrol  boats. The only change to the number of vessels is that the revised plans  include nine additional 25-foot short range boats and nine fewer 35-foot  long range boats than were included in the original plan. Table 2 compares  the number and types of Deepwater assets under the original and revised  plans.
section_title: Delivery Schedules for Deepwater Assets Have Changed
paragraphs: Estimated delivery schedules for the Deepwater assets have changed. For  some of the aircraft, deliveries have been projected for later years than  were estimated in the original plan. For example, the Coast Guard now  plans for delivery of its first 3 CN-235 Medium-Range Surveillance Aircraft  during calendar year 2008. Under the original plan, the Coast Guard had  anticipated delivery of the first 12 in 2006, with a total of 18 delivered by  the end of 2008. Final deliveries of the CN-235s under the 2006 revised  plan are now scheduled for 2027, as opposed to 2012 under the original  plan. According to the Coast Guard, the delivery schedule for the CN-235  Medium-Range Surveillance Aircraft was delayed because the Coast Guard  did not receive the anticipated level of funding in fiscal years 2002 and  2003, which required renegotiations. Figure 1 shows the original and  revised delivery schedules for Deepwater aircraft.
For vessels, the revised plans generally spread out deliveries of each class  of vessel over a larger number of years, as shown in figure 2. For example,  the original plan called for delivery of 58 of the 140-foot Fast Response  Cutters between 2018 and 2022. The revised plans call for delivering the  first Fast Response Cutter in 2007 or 2008, with additional cutters being  delivered every year from 2009 through 2027—a span of 21 years. The  Coast Guard originally planned to convert its legacy 110-foot patrol boats  to 123-foot patrol boats and, beginning in 2018, replace the 123-foot patrol  boats with 140-foot Fast Response Cutters. However, the patrol boat  conversion project was halted after the first 8 patrol boats because the  123-foot patrol boats could not meet post September 11, 2001 mission  requirements and were experiencing technical difficulties. Because of this,  the Coast Guard needed to advance the delivery of the Fast Response  Cutters.
section_title: Estimated Cost of Revised Deepwater Plans is $7 Billion Higher, Largely Reflecting Increased Homeland Security Mission Requirements
paragraphs: The total estimated cost of the revised Deepwater plans increased by   $7 billion over the original plan—from $17 billion to $24 billion. According  to the Coast Guard, most of the $7 billion increase is due to enhanced  homeland security mission requirements brought about by the events of  September 11, 2001. In particular, data provided by the Coast Guard show  that most of the $7 billion increase is attributable to costs for enhancing  and upgrading the capabilities of the planned Deepwater replacement  vessels. More specifically, as shown in table 3, upgrades to the Deepwater  vessels account for about $5.5 billion of the increase in the 2005 plan, and  $5.9 billion in the 2006 update.
Beyond the increases related solely to vessels, upgrades to the C4ISR and  maritime domain awareness capabilities to improve interoperability  between the Coast Guard and other Department of Homeland Security  components, as well as with the Department of Defense, account for the  second largest category of cost increases—increasing by $1.1 billion in the  2005 revised plan, and by $663 million in the 2006 plan. In contrast,  because the revised plans include upgrading the HC-130 aircraft and the  HH-60 helicopter rather than replacing them as called for in the original  plan and for scaling back on the number of unmanned aerial vehicles to be  acquired, costs for Deepwater aircraft decreased from the original plan to  the revised plan. Overall, costs for Deepwater aircraft were reduced by  about $600 million in the 2005 plan and by about $400 million in the 2006  plan from the amount included in the original plan.
According to the Coast Guard, the primary elements of the enhanced  homeland security mission requirements that contributed to the $7 billion  increase include the following:    Chemical, biological, and radiological detection and defense. For  this element, the additional capabilities included in the revised plans vary  by asset. The most extensive are for the National Security Cutter, which is  to have a sealed section within which crew can operate the ship in a  contaminated environment for limited time periods. In the event an area is  contaminated, such as from a terrorist attack, the crew can use radar,  heat-seeking sensors, and other equipment to determine what is  occurring—such as whether engines are operating, vessels are being  moved, or people are alive. Other Deepwater vessels and aircraft are to be  equipped with exposure suits and storage for those suits.
Antiterrorism and force protection. The revised plans call for more  powerful weapons for National Security Cutters, Offshore Patrol Cutters,  and Fast Response Cutters. Manual gun mounts on cutters will be replaced  with selected sensor-integrated, remote-operated, and semi-automated gun  systems. This weaponry is to give the Coast Guard enhanced capabilities  to protect its own cutters and other high value assets by, for example,  providing cutters with the ability to stop terrorists who have taken control  of a ship by disabling that ship’s propulsion with precision fire.
Airborne use of force and vertical insertion and delivery. The  revised plans call for the Deepwater helicopters to be fitted with weapons  and equipment that will enable armed teams to land on a vessel, such as in  the event a hostile group has taken over the vessel. Crew members can use  machine guns to provide cover while a team travels by rope from the  hovering helicopter to the vessel’s deck. Additionally, for certain terrorist  and criminal scenarios, the helicopter can use disabling fire to stop an  illegally operated boat. In the event of a terrorist attack and the right  circumstances, the disabling fire can be changed to deadly fire if necessary  to stop terrorists.
Interoperability with the Departments of Defense and Homeland  Security, as well as Rescue-21 equipment. All Deepwater vessels and  aircraft are to receive C4ISR enhancements that make them interoperable  with other DHS entities, DOD assets, and local first responders. These  enhancements include added circuits and equipment that provide full  voice communication and limited data communications between these  entities.
Extended/enhanced flight deck. The flight decks of the National  Security Cutter and Offshore Patrol Cutter are to be enlarged so that  helicopters from other Department of Homeland Security components and  from DOD can land on the cutters.
section_title: Deepwater Costs Could Rise if Funding Deviates from Levels Called For in the Plans
paragraphs: In May 2001, we reported that affordability was the biggest risk for the  Deepwater program because the Coast Guard’s contracting approach  depends on a sustained level of funding each fiscal year over the life of the  program. For the 2005 revised implementation plan, these funding levels  average over $1 billion per year and range from $650 million to over   $1.5 billion per year through fiscal year 2026. According to Coast Guard  officials, any significant or sustained deviation from the planned funding  levels would be costly to the Coast Guard in the short term and set off  ripples affecting the acquisition of Deepwater equipment for years to  come. The officials added that significant shortfalls would likely result in  increased costs, late delivery of equipment, and degradation of Deepwater  asset performance.
section_title: Model Used to Determine Revised Asset Mix Is Reliable, and the Coast Guard Hopes to Expand Its Use
paragraphs: In revising the Deepwater asset mix to meet new mission demands, the  Coast Guard undertook a series of analyses that used a computer  simulation model to project the operational effectiveness of a variety of  potential Deepwater force structures or asset mixes. We found that this  model contains reliable information and is useful for guiding decisions on  the revised Deepwater asset mix. Further, a Department of Defense review  board facilitated accreditation of the model and another group with  expertise in this type of modeling has studied the Coast Guard’s approach  and concluded that it is reliable. Through use of this model, the Coast  Guard projects that the Deepwater asset mix in the $24 billion revised  implementation plan will provide greater mission performance than the  asset mix in the original plan. Other factors beyond this model, such as  decisions of internal working groups and projected funding, also  contributed to the adoption of the revised Deepwater asset mix. Because  the model has proved useful for guiding Coast Guard decisions on the  proper asset mix for enhancing the mission performance of the Deepwater  assets, the Coast Guard is considering ways to expand the model to guide  decisions on meeting its Coast Guard-wide Government Performance and  Results Act (GPRA) performance goals.
subsections: section_title: Computer-Based Model Used in Analyzing Capacity Gaps Is Credible
paragraphs: After the events of September 11, 2001, the Coast Guard undertook a  series of analyses intended to determine what capability and capacity gaps  would exist if the asset mix in the original Deepwater plan were applied to  the revised Deepwater missions. To conduct this analysis, the Coast Guard  projected the performance of a variety of asset mixes using a computer- based operational effectiveness simulation model known as the Deepwater  Maritime Operational Effectiveness Simulation (DMOES). Using three  different capacity models, the Coast Guard generated three different  versions of the asset mix needed to meet Coast Guard performance  targets. The resulting force structures were then modeled in DMOES to  project their operational effectiveness. The results of this assessment led  the Coast Guard to change the asset mix for its revised Deepwater plan.
We found that DMOES, which provided important evidence for Deepwater  operational effectiveness analyses, contains reliable information for  decision making. Specifically, our review of various statistical aspects of  DMOES indicates that the parameters used in the DMOES model—the  targets, missions, weather events, and probability of target detection  present in the Deepwater environment—appear to be the result of a  thorough and rigorous process that enhanced the model’s credibility. In  performing our review of DMOES, we reviewed computer simulation  model criteria developed by an authority in the field of simulation  modeling and found that the DMOES model successfully addressed these  criteria. For example, the parameters used were derived from historical  events (e.g., target detection or weather events), which helped satisfy the  criterion that interactions between the modeled system and the outside  environment be considered. To ensure use of valid and current data for its  major updates of DMOES, Coast Guard gathered updated historical data  and compared these data to data from past events. Further, because the  Coast Guard modeled target detection capabilities for the assets at less  than their full potential, the asset’s target detection capabilities do not  appear to be overstated.
In addition, independent authorities, in their reviews of DMOES, have  assessed the model and have accredited it for force structure planning. For  example, the MITRE Corporation, in an independent analysis of the  performance gap analysis process (of which DMOES was a key  component), found that the process and the resulting analytic results were  “likely the most complete and comprehensive campaign-level study  conducted by any uniformed service in recent times.” Further, the Coast  Guard submitted DMOES to a verification, validation, and accreditation  review monitored and facilitated by the Joint Accreditation Support  Activity. The DMOES Accreditation Review Board, consisting of Coast  Guard officials and external experts in the field of military force structure  determinations and capability-based planning, conducted the actual  review and accredited the DMOES model for acquisition support and force  structure planning.
section_title: Other Factors Also Affected Revised Deepwater Asset Mix
paragraphs: While the capability and capacity gaps identified in the performance gap  analysis process were a key input into the decisions leading to the revised  Deepwater asset mix, they were not the only factor. The Coast Guard also  shaped the Deepwater asset mix based on budget considerations and  information developed by an internal working group. In particular:     Coast Guard officials stated that affordability was a key factor in shaping  the revised Deepwater asset mix. According to the officials, Deepwater  was never intended to be an unconstrained acquisition program, and the  $24 billion force structure was determined through a process of modeling  performance of anticipated asset mixes, weighed against expected funding  levels over the life of the program, to come up with an optimal balance of  performance and affordability. As a result, the revised Deepwater asset  mix was developed to maximize the system’s capabilities and capacities  within this $24 billion budget. The officials added that while the $24 billion  budget may not allow for all desired capabilities on each asset, capabilities  are being designed for later refit, if applicable.
Further, in April 2004, the Assistant Commandant for Operations  Capability commissioned an Aviation Legacy Alternatives Working Group  to analyze possible alternatives to the aviation force structure in the  original Deepwater plan. This working group provided key data used to  enhance the performance gap analysis process. For example, as a result of  the working group’s analyses, the Coast Guard decided to convert and  upgrade two of its four legacy aircraft (HC-130 and HH-60) and replace  only the HU-25. This strategy was deemed by the Coast Guard to be the  most cost-effective solution for meeting Deepwater mission requirements.
According to the Coast Guard, other alternatives added significant  capacity, but at a greater cost.
section_title: Model Indicates That the Revised Asset Mix Will Provide Improved Mission Performance over the Original Plan
paragraphs: The most recent DMOES runs conducted by the Coast Guard, published in  October 2005, project that the revised Deepwater asset mix will provide “a  significant improvement in traditional Coast Guard mission performance”  compared to the original Deepwater asset mix. This marked the first time  that the Coast Guard used DMOES to model the operational effectiveness  of the revised Deepwater asset mix. According to the Coast Guard, the  projected improvement in the overall mission performance of the revised  asset mix is due mainly to increased maritime surveillance aircraft and,  more specifically, to the converted HC-130, which will be present in  greater numbers and with greater capabilities than the comparable long- range surveillance aircraft from the original Deepwater plan. Table 4  provides a brief summary of the results of our analysis of the latest  DMOES modeling in terms of how the revised asset mix is projected to  improve performance for the Coast Guard’s various Deepwater missions.  Appendix I provides more details on our analysis of the latest DMOES  modeling.
section_title: Coast Guard Is Exploring Options for Applying Further Modeling for Projecting Coast Guard- wide Performance Capabilities
paragraphs: Though DMOES was an accredited, rigorous simulation model effective in  supporting Deepwater force structure planning, it does not capture the  impact of non-Deepwater asset contributions and, therefore, does not  provide a means for the Coast Guard to estimate the extent to which its  entire fleet of aircraft and vessels will allow it to meet Coast Guard-wide  GPRA performance targets. The Coast Guard is aware of this limitation  and is exploring options for expanding DMOES to encompass all Coast  Guard assets—both Deepwater and non-Deepwater—in an effort to  provide for a true analysis of Coast Guard-wide mission performance  capabilities. While this has not yet occurred, Coast Guard officials told us  they were reasonably confident that the cumulative effect of merging the  revised Deepwater assets with its non-Deepwater assets would allow the  Coast Guard to meet GPRA targets for those missions involving Deepwater  aircraft and vessels. In the interim, the Coast Guard has taken steps to  measure the impact of Deepwater assets on Deepwater-related metrics.  Since 2002, the Coast Guard has annually reviewed—and plans to continue  reviewing—the most recent complete year’s worth of data and estimates  the Deepwater-only contribution toward meeting performance goals for  seven particular performance metrics. These performance metrics and  results for the most recent year available are shown in table 5.
Disaggregating performance data to reflect Deepwater-only contributions  provides an estimate of the extent to which the Deepwater fleet is helping  the Coast Guard meet these key targets on an annual basis. For example,  as a result of these efforts, the Coast Guard determined that Deepwater  assets saved 92.6 percent of lives at risk after Coast Guard notification in  fiscal year 2004, which is slightly below the Deepwater asset target value  of 93 percent.
section_title: Progress Continues in Making Recommended Improvements
paragraphs: Our past concerns about the Deepwater program have been in three main  areas—ensuring better program management and contractor oversight,  ensuring greater accountability on the part of the system integrator, and  creating sufficient competition to help act as a control on costs—and we  made a total of 11 recommendations to address these concerns. During  our 2005 review, we determined that the Coast Guard had addressed and  fully implemented 2 of these 11 recommendations. The Coast Guard  disagreed with and declined to implement a separate recommendation that  pertained to updating its cost baseline to determine whether the  Deepwater acquisition approach is costing more than a conventional  acquisition approach. While we stand behind our original  recommendation, we decided not to pursue it further because the Coast  Guard determined that the cost to implement this recommendation was  excessive. Thus, at the time we began our current review, 8 of the   11 recommendations were not yet fully implemented. On the basis of  information we gathered for this review, we consider 3 of these   8 recommendations to be fully implemented. The Coast Guard is in the  process of taking actions to implement 3 more recommendations, but full  implementation is dependent on seeing results or completion of actions  that are not yet in final form. The 2 remaining recommendations, both  relating to overall program management and oversight, remain  problematic. One relates to improving the effectiveness of integrated  product teams, the other to providing field personnel with guidance and  training on transitioning to new Deepwater assets. In each case, the Coast  Guard has taken actions, but our review of program reports and our  discussions with program and field personnel indicate the problems still  remain. In all cases, however, the steps needed to fully implement these  recommendations seem relatively clear.
Table 6 provides an overview of the 11 recommendations. The sections  below discuss the recommendations made in each of the three areas of  concern, describing the initial issue that led to the recommendation, the  steps taken to date to address it, and our rationale for considering the  recommendation as being fully implemented or not. Where we make a  determination that a recommendation has not yet been implemented, we  indicate what actions are needed.
subsections: section_title: Coast Guard’s Efforts to Improve Oversight and Program Management Show Mixed Results
paragraphs: We continue to see mixed results in the Coast Guard’s efforts to improve  oversight and management of the Deepwater program. The Coast Guard  has put in place a human capital plan to help ensure adequate staffing of  the Deepwater program and has taken actions to improve the effectiveness  of integrated product teams. However, subcontractor collaboration and  provision of guidance on transitioning to new Deepwater assets to field  personnel, particularly as it pertains to maintenance and logistics  responsibilities, continue to need additional attention.
section_title: Put in Place a Human Capital Plan to Ensure Adequate Staffing of the Deepwater Program
paragraphs: Original issue: As early as 2001, we noted that difficult human capital  challenges would need to be addressed, including the need to recruit and  train sufficient staff to manage and oversee the Deepwater contract.  Reviewing this matter again in 2004, we found that the Coast Guard had  not funded the number of staff requested by the Deepwater program and  had not adhered to the processes outlined in its human capital plan for  addressing turnover of Deepwater officials, particularly Coast Guard  personnel. These staffing shortfalls contributed to problems in making  timely decisions and keeping pace with the workload.
Steps taken: The Coast Guard took several steps to address this issue. Its  initial steps involved hiring contractors to assist with program support  functions, shifting some positions from being staffed by military personnel  to civilian personnel to mitigate turnover risk, and identifying the hard-to- fill positions and developing recruitment plans specifically for them.  Subsequent to these changes, the Deepwater program’s executive officer  (1) approved a revised human capital plan in February 2005 emphasizing  workforce planning and (2) is developing ways to leverage institutional  knowledge as staff rotate out of the Deepwater program. The Coast Guard  plans to review the human capital plan annually to ensure continual  alignment between human capital management and actual program  performance. The Coast Guard has also placed added emphasis on staffing  when formulating the program’s budget request—for example, in adding  contracting officers and specialists. Finally, the Coast Guard has worked  closely with the Department of Homeland Security and the Defense  Acquisition University to provide training for Deepwater personnel.
Recommendation status: The steps the Coast Guard has taken appear  sufficient to address matters related to adequately staffing the Deepwater  program and mitigating turnover, and therefore we consider this  recommendation to be fully implemented.
subsections: section_title: Strengthening Integrated Product Teams
paragraphs: Original issue: Effective management of the Deepwater program depends  heavily on strong collaboration among the Coast Guard, the system  integrator, and the subcontractors. Integrated product teams (IPTs), the  Coast Guard’s primary tool for managing the Deepwater program,  overseeing contractor activities, and ensuring collaboration have  experienced difficulty from the outset. IPTs, which are generally chaired  by a subcontractor representative and consist of members representing  the subcontractors and the Coast Guard, are responsible for overall  program planning and management, asset integration, and overseeing  delivery of specific Deepwater assets. In 2004, we reported these teams  were struggling to carry out their missions because of four major issues:  lack of timely charters to provide authority needed for decision making,  inadequate communication among team members, high turnover, and  insufficient training.
Steps taken: In 2005, we found that all IPTs had charters and their  members had received entry-level training. Decision making, however,  continued to be largely compartmented. Since then, the Coast Guard has  established domain management teams to serve as oversight and conflict  resolution entities for the IPTs. According to Coast Guard officials, these  teams are also to enhance collaboration on issues that cut across several  IPTs. Monthly assessments show IPTs have continued to improve their  effectiveness across all performance measures.
Recommendation status: While the Coast Guard has taken some actions,  we do not believe the actions are sufficient to consider the  recommendation to be fully implemented because there are indications  that collaboration among subcontractors remains inconsistent. Last year  we pointed out that ICGS’s two major subcontractors, Lockheed Martin  and Northrop Grumman, were operating under their own management  systems and that this approach could lessen the likelihood that a system- of-systems outcome would be successfully achieved. During our current  review, Coast Guard performance monitors and the program’s executive  officer reported that collaboration among the subcontractors continues to  be problematic and that ICGS wields little influence to compel decisions  among them. For example, when dealing with proposed design changes to  assets under construction, ICGS submits the changes as two separate  proposals from both first-tier subcontractors rather than coordinating the  separate proposals into one coherent plan. According to Coast Guard  performance monitors, this approach complicates the Coast Guard’s  review of the needed design change because the two proposals often carry  overlapping work items, thereby forcing the Coast Guard to act as the  system integrator in these situations.
section_title: Providing Field Personnel with Guidance and Training on Transitioning to New Deepwater Assets
paragraphs: Original issue: In 2004, we found the Coast Guard had not effectively  communicated decisions on (1) how new Deepwater and existing assets  are to be integrated during the transition and (2) whether Coast Guard or  contractor personnel (or a combination of the two) will be responsible for  maintenance of the Deepwater assets. For example, Coast Guard field  personnel, including senior-level operators and naval engineering support  command officials, said they had not received information about how they  would be able to continue accomplishing their missions using existing  assets while also being trained on the new assets.
Steps taken: The Coast Guard has taken some steps to improve the level  of communication between the Deepwater program and field operators  and maintenance personnel. A November 2004 analysis of the Deepwater  program’s communication process, conducted in coordination with the  National Graduate School, found that the communication and feedback  process was inadequate. Since then, the Coast Guard has placed more  emphasis on outreach to field personnel, including surveys, face-to-face  meetings, and presentations. More recently, officials from the Atlantic and  Pacific Area Commands, Maintenance and Logistics Commands, and the  Aircraft Repair and Supply Center agreed that Deepwater program  officials have significantly improved the frequency and types of  information flowing from the program office to the field. In addition, field  personnel are members of several IPTs and working groups, and ICGS has  placed liaisons at several field locations.
Recommendation status: While the Coast Guard has taken some actions,  there are indications that the actions are not yet sufficient to consider the  recommendation to be fully implemented. In particular, our review of  relevant documents and our discussions with key personnel make clear  that field operators and maintenance personnel are still concerned that  their view are not adequately acknowledged and addressed, and have little  information about maintenance and logistics plans for the new Deepwater  assets. For example, though the first National Security Cutter is to be  delivered in August 2007, field and maintenance officials have yet to  receive information on plans for crew training, necessary shore facility  modifications, or how maintenance and logistics responsibilities will be  divided between the Coast Guard and ICGS. According to Coast Guard  officials, many of these decisions need to be made and communicated very  soon in order to allow for proper planning and preparation in advance of  the cutter’s delivery.
section_title: More Time Needed to Determine Adequacy of Steps Taken to Improve System Integrator Accountability
paragraphs: Unlike actions on the previous recommendations, Coast Guard actions to  provide better input from Coast Guard performance monitors and to hold  the system integrator more accountable for performance appear to be  largely sufficient. We cannot determine whether the Coast Guard has  implemented several of our recommendations in this area, however, until  more Deepwater assets are delivered and the results of these actions can  be assessed.
subsections: section_title: Providing Better Input from Coast Guard Performance Monitors
paragraphs: Original issue: In 2004, we reported that the Coast Guard’s award fee  evaluation of the first year of ICGS’s performance was based on  unsupported calculations and relied heavily on subjective judgments.  Rating procedures used by Coast Guard performance monitors were  inconsistent, as were procedures for calculating scores, leading to  questions about whether the award fee decision was well supported.
Actions taken: The Coast Guard has provided additional guidance and  training to performance monitors, better allowing them to link their  comments with specific examples within their respective areas of  responsibility. The Coast Guard has also improved the consistency of the  format that performance monitors use to provide input about the system  integrator’s performance and revised assessment criteria to more clearly  differentiate between objective measures (that is, developed using  automated tools and compared against defined standards) and subjective  evaluations. Weights have been assigned to each set of evaluation factors,  and the Coast Guard continues to adjust these factors to achieve an  appropriate balance between automated results and eyewitness  observations.
Recommendation status: The Coast Guard’s efforts to provide better  guidance and training, improve the consistency of the format for  performance monitors’ input, and clarify performance assessment criteria  appear sufficient for addressing the issue, and therefore we consider this  recommendation to be fully implemented.
section_title: Holding the System Integrator Accountable for Improving Effectiveness of the Integrated Product Teams
paragraphs: Original issue: In 2004, we found that the system integrator, whose  subcontractors chaired the IPT working groups, was not being held  accountable for IPT effectiveness in its performance assessments. Actions  taken: The Coast Guard changed award fee measures to place additional  emphasis on the system integrator’s responsibility for making the IPTs  effective. Award fee criteria now incorporate the administration,  management commitment, collaboration, training, and empowerment of  these teams.
Recommendation status: With IPTs’ performance now included in the  criteria for measuring the system integrator’s performance, we consider  this recommendation to be fully implemented.
section_title: Establishing a Time Frame for Putting Steps in Place to Measure Contractor’s Progress toward Improving Operational Effectiveness
paragraphs: Original issue: In 2001, the Coast Guard set a goal of developing  measures, within 1 year after contract award, to conduct annual  assessments of the system integrator’s progress toward achieving the  three overarching goals of the Deepwater program, including increased  operational effectiveness. In 2004, we found that the time frame for the  first review of the contractor’s performance against the Deepwater goals  had slipped. The former Deepwater chief contracting officer told us that  he anticipated that the metrics would be in place in the fourth year of the  contract, the same year the Coast Guard would decide whether or not to  extend the contract.
Steps taken: The Coast Guard has since developed modeling  capabilities—namely the DMOES model discussed earlier—to simulate the  effect of the new assets’ capabilities on the Coast Guard’s ability to meet  its missions. Coast Guard officials told us that they are now beginning to  track the operational effectiveness of the Deepwater program using both  the DMOES model and actual mission performance data. Further, at the  Coast Guard’s request, the Center for Naval Analyses developed a tool to  measure the “presence” of Deepwater assets—that is the number of square  miles of ocean in which Deepwater aircraft and vessels can detect,  identify, and prosecute targets. In addition, Coast Guard officials have also  begun using mission performance data from 2004, the most recent year of  complete information, to measure the contribution provided by Deepwater  systems or assets in seven mission areas: search and rescue, cocaine  seizure rate, illegal or undocumented migrant interdiction, foreign fishing  vessel interdiction, protection of living marine resources, national  defense/military readiness, and international ice patrol. Coast Guard  officials acknowledge that this is difficult, though, because the data on  mission results and accomplishments do not differentiate between  Deepwater assets and non-Deepwater assets. Coast Guard officials said  doing so should become easier as more Deepwater assets come on line  and as analytical tools are refined.
Recommendation status: Although the models have been developed and  are being refined to measure operational effectiveness, there are too few  Deepwater assets currently in operation to effectively measure the system  integrator’s actual performance in improving operational effectiveness. As  a result, we do not consider this recommendation to be fully implemented.
We recognize, though, that as more Deepwater assets and systems come  on line, the amount of data will increase and the analytical tools will be  more refined so that the Coast Guard should be in a better position to   (1) discern the Deepwater program’s contribution to operational  effectiveness and (2) fully implement this recommendation.
section_title: Establishing Criteria to Determine When to Adjust the Project Baseline and Document the Reasons for Change
paragraphs: Original issue: Establishing a solid baseline against which to measure  progress in lowering total ownership cost (TOC) is critical to holding the  system integrator accountable. However, during our 2004 review, we  found that the Coast Guard’s Deepwater TOC baseline had been  significantly changed from what had been originally envisioned and that  further changes could be made as a result of variables such as fuel costs or  vessels’ operating tempo. At the time, Coast Guard officials explained that  proposed changes to the baseline would be approved by the program  executive officer on a case-by-case basis, though the Coast Guard had not  developed criteria for potential upward or downward adjustments to the  baseline.
Steps taken: In response to our concerns, the Coast Guard began using  criteria from its Major Systems Acquisition Manual as the basis for  adjusting the TOC baseline. These criteria allow the baseline to be  adjusted based on significant changes in mission requirements, schedule  changes, or project funding, or for specific congressional actions. Coast  Guard officials also told us that they have also added criteria for making  changes to the baseline, such as:  insufficient program funding or inflationary pressure that exceeds the  assumptions and   natural disasters or periods of national emergency that require a deviation  from the baseline’s cost, schedule, or performance parameters.
Coast Guard officials said that approval of revisions to the program’s  overall baseline must come through approved decision memorandums  from the Agency Acquisition Executive, who is the Vice Commandant of  the Coast Guard. The Deepwater Program Executive Officer still has  authority to approve baseline revisions at the asset and domain level.  Depending on their severity, baseline changes now are also subject to  review and approval by the Department of Homeland Security (DHS), the  Coast Guard’s parent agency. The Coast Guard is required to submit  Deepwater program baseline information to DHS on a quarterly basis, and  the project is subject to an annual review by the DHS Investment Review  Board. According to DHS officials, a baseline breach of 8 percent or more  would require that the Coast Guard provide information on the causal  factors and propose corrective actions to rectify the breach. The officials  added that, if the baseline breach is considered significant, the Office of  Management and Budget is to be notified that the program will have to  undergo a rebaselining and its funding profile will need to be altered.  Further, as a result of its latest review of the Deepwater program, the DHS  Investment Review Board has asked that, in addition to overall program  baseline information, the Coast Guard also provide baseline information  for each of the Deepwater assets. This will provide DHS with more insight  into the program’s cost, schedule, and performance.
Recommendation status: The Coast Guard’s steps, combined with DHS’s  oversight requirements, should be sufficient to resolve this issue. At  present, however, DHS’s policy directive is only in draft form. We will  consider this recommendation to be fully implemented when the  management directive is finalized.
section_title: Effects of Steps Taken to Control Future Costs through Competition Will Take Time to Assess
paragraphs: The Coast Guard has taken a number of actions to address the remaining  recommendation in this area, which relates to holding the system  integrator accountable for ensuring competition among subcontractors.  However, until the effects of these actions are more apparent, we are not  able to consider the recommendation as being implemented.
subsections: section_title: Developing a Plan for Holding the System Integrator Accountable for Ensuring Adequate Competition among Suppliers
paragraphs: Original issue: Competition is a key component for controlling costs in  the Deepwater program and a guiding principle for DHS’s major  acquisitions. In 2004, we found that beyond the initial 5-year contract  period, the Coast Guard had no way to ensure competition was occurring  because it did not have mechanisms in place to measure the extent of  competition or to hold the system integrator accountable for steps taken  to achieve competition. Shortly before, the system integrator had adopted  Lockheed Martin’s “open business model” as a corporate policy to help  ensure competition and keep costs under control. However, the open  business model is not a formal policy involving specific decision points to  ensure that competition will be considered. Further, the first-tier  subcontractors, Lockheed Martin and Northrop Grumman, have largely  continued to follow their own procurement procedures and guidance for  determining whether competition will occur and the suppliers who will be  invited to compete for Deepwater assets.
Steps taken: To address our recommendation about ensuring out-year  competition among second-tier suppliers, the Coast Guard contracted with  Acquisitions Solutions, Inc. (ASI), to assess the amount of second-tier  competition conducted by ICGS during 2004. ASI issued a report in May  2005 that, among other things, found that the open business model had not  been fully embraced by Northrop Grumman despite its being an ICGS  corporate policy. The report made nine recommendations aimed at  improving competition throughout the Deepwater program. According to  Deepwater officials, ICGS developed a plan to adopt all nine  recommendations by March 1, 2006, and is providing training on use of the  open business model to Northrop Grumman personnel working on the  Deepwater program. Further, Coast Guard officials reported that  competition will be assessed as a part of the award fee assessment  subjective criteria for the fifth year of the contract and the Coast Guard  will specifically examine the system integrator’s ability to control costs by  assessing the degree to which competition is fostered at the major  subcontractor level during the award term decision process later this  year.
Recommendation status: While steps already under way appear to be  sufficient to resolve our concerns, we cannot consider this  recommendation as being fully implemented until the Coast Guard has  addressed the ASI recommendations and results of the next award term  assessment are known.
section_title: Concluding Observations
paragraphs: The Coast Guard has done a commendable job of adapting the Deepwater  program to post-September 11 realities. Our analysis shows that Coast  Guard officials used sound analytical methods to assess the revised needs  for aircraft and vessels. Coast Guard officials have also taken strong  efforts to address concerns about program management and contract  performance and have largely implemented or are in the process of  implementing steps that would help mitigate these concerns. We agree  that the Coast Guard would be well served to continue developing ways to  use its computer modeling to establish clear relationships between its mix  of assets—aircraft and vessels—and its Deepwater and agency-level  performance goals. We have pointed out in past reports that the Coast  Guard lacks clear measures of how its resources are linked to achieving  performance goals, so these steps should help resolve this concern. We  realize that this ongoing effort will likely take some time to complete.  While the Coast Guard has made good progress in addressing our  recommendations, there are aspects of the Deepwater program that will  require continued attention. First, the Deepwater program continues to  face a degree of underlying risk, in part because of the unique approach  involving a system-of-systems approach with the contractor acting as  overall integrator, and in part because it so heavily tied to precise year-to- year funding requirements over the next two decades. Further, a project of  this magnitude will likely continue to experience other concerns and  challenges beyond those that have emerged so far. It will be important for  Coast Guard managers to continue careful monitoring of contractor  performance and to continue addressing program management concerns  as they arise.
section_title: Agency Comments
paragraphs: We requested comments on a draft of this report from the Department of  Homeland Security and the U.S. Coast Guard. The U.S. Coast Guard  provided technical comments, which have been incorporated into the  report as appropriate.
We are providing copies of this report to the Secretary of the Department  of Homeland Security, the Commandant of the U.S. Coast Guard, and  interested congressional committees. The report will also be made  available to others upon request. In addition, the report will be available at  no charge on GAO’s Web site at http//www.gao.gov.
If you or your staff have any questions about this report, please contact me  on (415) 904-2200 or wrightsonm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix II.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report, which focuses on the Coast Guard’s Deepwater management  challenges, provides details on three issues: (1) a comparison of the  revised Deepwater implementation plan issued in August 2005 with the  original (August 2002) plan in terms of cost, time frames, and the balance  of legacy and replacement assets; (2) an assessment of the degree to  which the operational effectiveness model and other analytical methods  used by the Coast Guard to develop the revised Deepwater asset mix are  sound and appropriate for such a purpose; and (3) an assessment of the  progress made in implementing our prior recommendations regarding  Deepwater program management.
To compare the revised Deepwater implementation plans issued in August  2005 and February 2006 with the original (August 2002) Deepwater  implementation plan in terms of cost, time frames, and the balance of  legacy and replacement assets, we analyzed the original and revised  Deepwater implementation plans and related guidance. We also reviewed  and analyzed relevant Coast Guard documentation on changes in missions,  costs, asset mix, asset capabilities, and asset delivery schedules. We  supplemented the documentation reviews and analyses with discussions  with officials from the Deepwater Program Executive Office. Finally, we  discussed the risks associated with the Deepwater program’s reliance on a  sustained level of funding through 2027 and the implications of these risks.
To assess the degree to which the operational effectiveness models and  other analytical methods used to develop the revised Deepwater asset mix  are sound and appropriate for such a purpose, we reviewed the capacity  and operational effectiveness models used in determining the current  Deepwater asset mix to ensure that the approach was sound and that  appropriate assumptions were made in the models’ use. This review  involved assessing Coast Guard documentation on how its models were  developed and executed, determining the views of knowledgeable  independent parties on the Coast Guard’s operational effectiveness model,  and interviewing cognizant Coast Guard officials. These interviews also  included discussions of how these models, and other factors, were used in  developing the current Deepwater asset mix, as well as whether the Coast  Guard has developed an approach for determining the extent to which the  Deepwater asset mix will allow it to meet its performance targets.
In assessing the Coast Guard’s modeling and other analytical methods  used for developing the revised Deepwater asset mix, we paid particular  attention to the most recent performance gap analysis (PGA) study (PGA  IV), which compared the projected performance of the revised Deepwater  asset mix to that of the original Deepwater asset mix—so that we could  gain a better understanding of how these results were used in developing  the revised Deepwater asset mix. Specifically, we reviewed the report’s  methodology and requested additional clarifying information to help  determine if the analytic work supported the report’s conclusions. As part  of our assessment, we developed an analysis that departs from what the  Coast Guard describes in its report in two important ways. First, and most  important, the Coast Guard assigned a linear scale ranging from 1 to 5 to  five statistical categories describing the distribution of the analysis data.  This range assigned numerical values to the degree to which the revised  asset mix was projected to outperform (or underperform) the original  asset mix, with 1 representing projected performance two or more  standard deviations below that of the original asset mix, up to 5,  representing projected performance two or more standard deviations  above that of the original asset mix. It is our opinion that this type of linear  scale is not appropriate for capturing the variations in projected  performance. Accordingly, we used a weighting scheme for these  categories (known as z-scores) that better reflects the relationship among  these categories. The z-scores take into account the statistical property  that being two standard deviations away from the mean value is almost  five times more difficult than being one standard deviation away from the  mean. Second, we compared our calculated performance measure weights  to a standard in order to assess if our weighting scheme would affect the  study’s conclusions. Since the methodology identified three mission  significance categories and four regional mission priority categories, we  compared our recalculated weights based upon the z-score with the  weights we would expect to see if all mission performance measures  across all mission priorities for the four modeled regions had exceed one  standard deviation above the mean in improvement. Despite the different  methodologies used, our results generally aligned with what the Coast  Guard reported in PGA IV.
To determine the status of the Coast Guard’s implementation of our prior  recommendations for improving program management, strengthening  contractor accountability, and controlling costs, we reviewed and  analyzed briefings and relevant documentation provided by the Deepwater  Program Executive Office on actions taken to address our concerns. We  reviewed and analyzed documentation on the Coast Guard’s assessment of  the contractor’s system integration and management performance in the  first period of the fourth year of the contract, including written comments  by the performance monitors. We also reviewed and analyzed information  on Deepwater integrated product teams, including membership lists and  briefings provided by the Coast Guard on measures of effectiveness for the  teams. We analyzed the Coast Guard’s plans to increase communications  to field operators, and documentation from field operators and  maintenance personnel regarding these communications. Further, we  analyzed the February 2005 Deepwater revised Human Capital Plan to  identify changes that have been made and discussed Deepwater Program  Office staffing plans with Coast Guard officials. To supplement our  analyses of the relevant documentation, we held several meetings with the  Deepwater Program Executive Officer, the Deputy Program Executive  Officer, and a number of Deepwater staff, including contracting officials  and representatives from the system integrator. We also held discussions  with Coast Guard Deepwater performance monitors to discuss their  written comments to the latest award fee assessment. We also held  discussions with officials from the Pacific Area Command and Pacific Area  Maintenance and Logistics Command in Alameda, California; the Atlantic  Area Command and Atlantic Area Maintenance and Logistics Command in  Norfolk, Virginia; and the Aircraft Repair and Supply Center in Elizabeth  City, North Carolina. Further, we reviewed acquisition guidance and spoke  with officials from the Department of Homeland Security regarding their  oversight of the Deepwater acquisition program baseline.
We performed our review from August 2005 to March 2006 in accordance  with generally accepted government auditing standards.
section_title: Appendix II: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Steven Calvo,   Christopher Conrad, Adam Couvillion, Christine Davis, Art James,   Julie Leetch, Michele Mackin, Stan Stenersen, and Linda Kay Willard made  key contributions to this report.
section_title: GAO Related Products
paragraphs: Coast Guard’s Acquisition Management: Deepwater Project’s  Justification and Affordability Need to Be Addressed More Thoroughly,  GAO/RCED-99-6 (Washington, D.C.: Oct. 26, 1998).
Coast Guard: Budget Challenges for 2001 and Beyond,   GAO/T-RCED-00-103 (Washington, D.C.: March 15, 2000).
Coast Guard: Progress Being Made on Deepwater Project, but Risk  Remain, GAO-01-564 (Washington, D.C.: May 2, 2001).
Coast Guard: Actions Needed to Mitigate Deepwater Project Risks,   GAO-01-659T (Washington, D.C.: May 3, 2001).
Coast Guard: Strategy Needed for Setting and Monitoring Levels of  Effort for All Missions, GAO-03-155 (Washington, D.C.: Nov. 12, 2002).
Coast Guard: Comprehensive Blueprint Needed to Balance and Monitor  Resource Use and Measure Performance for All Missions, GAO-03-544T  (Washington, D.C.: March 12, 2003).
Coast Guard: Challenges during the Transition to the Department of  Homeland Security, GAO-03-594T (Washington, D.C.: April 1, 2003).
Contract Management: Coast Guard’s Deepwater Program Needs  Increased Attention to Management and Contractor Oversight,   GAO-04-380 (Washington, D.C.: March 9, 2004).
Coast Guard: Replacement of HH-65 Helicopter Engine, GAO-04-595  (Washington, D.C.: March 24, 2004).
Coast Guard: Key Management and Budget Challenges for Fiscal Year  2005 and Beyond, GAO-04-636T (Washington, D.C.: April 7, 2004).
Coast Guard: Deepwater Program Acquisition Schedule Update Needed,  GAO-04-695 (Washington, D.C.: June 14, 2004).
Coast Guard: Observations and Agency Priorities in Fiscal Year 2006  Budget Request, GAO-05-364T (Washington, D.C.: March 17, 2005).
Coast Guard: Preliminary Observations on the Condition of Deepwater  Legacy Assets and Acquisition Management Challenges, GAO-05-307T  (Washington, D.C.: April 20, 2005).
Coast Guard: Preliminary Observations on the Condition of Deepwater  Legacy Assets and Acquisition Management Challenges, GAO-05-651T  (Washington, D.C.: June 21, 2005).
Coast Guard: Progress Being Made on Addressing Deepwater Legacy  Asset Condition Issues and Program Management, but Acquisition  Challenges Remain, GAO-05-757 (Washington, D.C.: July 22, 2005).