Coast Guard: Preliminary Observations on the Condition of Deepwater Legacy Assets and Acquisition Management Challenges
Released: 2005-06-21T00:00:00

Letter

Background

As the lead federal agency for maritime homeland security within the Department of Homeland Security, the Coast Guard is responsible for homeland and nonhomeland security missions, including ensuring security in ports and waterways and along coastlines, conducting search and rescue missions, interdicting drug shipments and illegal aliens, enforcing fisheries laws, and responding to reports of pollution. The deepwater fleet, which consists of 186 aircraft and 88 cutters of various sizes and capabilities, plays a critical role in all of these missions. As shown in table 1, the fleet includes fixed-wing aircraft, helicopters, and cutters of varying lengths.

Some Coast Guard deepwater cutters were built in the 1960s. Notwithstanding extensive overhauls and other upgrades, a number of the cutters are nearing the end of their estimated service lives. Similarly, while a number of the deepwater legacy aircraft have received upgrades in engines, operating systems, and sensor equipment since they were originally built, they too have limitations in their operating capabilities.

In 1996, the Coast Guard began developing what came to be known as the Integrated Deepwater System acquisition program as its major effort to replace or modernize these aircraft and cutters. This Deepwater program is designed to replace some assets—such as deteriorating cutters—with new cutters and upgrade other assets—such as some types of helicopters—so they can meet new performance requirements.

The Deepwater program represents a unique approach to a major acquisition in that the Coast Guard is relying on a prime contractor—the system integrator—to identify and deliver the assets needed to meet a set of mission requirements the Coast Guard has specified. In 2002, the Coast Guard awarded a contract to Integrated Coast Guard Systems (ICGS) as the system integrator for the Deepwater program. ICGS has two main subcontractors—Lockheed Martin and Northrop Grumman—that in turn contract with other subcontractors. The resulting program is designed to provide an improved, integrated system of aircraft, cutters, and unmanned aerial vehicles to be linked effectively through systems that provide command, control, communications, computer, intelligence, surveillance, reconnaissance, and supporting logistics. We have been reviewing the Deepwater program for several years. In recent reports we have pointed out difficulties the Coast Guard has been having in managing the Deepwater program and ensuring that the acquisition schedule is up to date and on schedule.

The existing schedule calls for acquisition of new assets under the Coast Guard’s Deepwater program to occur over an approximately 20-year period. By 2007, for example, the Coast Guard is to receive the first National Security Cutter, which will have the capability to conduct military missions related to homeland security. Plans call for 6 to 8 of these cutters to replace the 12 existing 378-foot cutters. However, in order to carry out its mission effectively, the Coast Guard will also need to keep all of the deepwater legacy assets operational until they can be replaced or upgraded.

Deepwater Legacy Assets Show General Decline in Condition, but Current Measures Do Not Capture True Extent

Coast Guard condition measures show that the deepwater legacy assets generally declined between 2000 and 2004, but the Coast Guard’s available condition measures are inadequate to capture the full extent of the decline in the condition of deepwater assets with any degree of precision. Other evidence we gathered, such as information from discussions with maintenance personnel, point to conditions that may be more severe than the available measures indicate. The Coast Guard acknowledges that it needs better condition measures but has not yet finalized or implemented such measures.

Coast Guard’s Condition Measures Show General Decline in Deepwater Assets, with Some Fluctuations

During fiscal years 2000 through 2004, the Coast Guard’s various condition measures show a general decline, although there were year-to-year fluctuations (see table 2). For deepwater legacy aircraft, a key summary measure of the condition—the availability index (the percentage of time aircraft are available to perform their missions)—showed that except for the HU-25 medium-range surveillance aircraft, the assets continued to perform close to or above fleet availability standards over the 5-year period. In contrast, other condition measures for aircraft, such as cost per flight hour and labor hours per flight hour, generally reflected some deterioration. For cutters, a key summary measure of condition—percent of time free of major casualties—fluctuated but generally remained well below target levels. The number of major casualties generally rose from fiscal years 2000 through 2003, and then dropped slightly in fiscal year 2004.

Another, albeit less direct, measure of an asset’s condition is deferred maintenance—the amount of scheduled maintenance that must be postponed on an asset in order to pay for unscheduled repairs. Such deferrals can occur when the Coast Guard does not have enough money to absorb unexpected maintenance expenditures and still perform all of its scheduled maintenance, thus creating a backlog. For example, in spring 2004, while on a counter-drug mission, the 210-foot cutter Active experienced problems in the condition of its flight deck that were to be corrected during its scheduled depot-level maintenance. However, because of a shortage of maintenance funds, the maintenance was deferred and the flight deck not repaired. As a result, the cutter lost 50 percent of its patrol time, since the required support helicopters could not take off from or land on it.

As table 3 shows, deferred maintenance does not show a clear pattern across all classes of deepwater legacy assets. For the deepwater legacy aircraft, the overall amount of estimated deferred maintenance increased each year during fiscal years 2002 through 2004, from $12.3 million to about $24.6 million. However, most of the increase came for one type of asset, the HH-60 helicopter, and the increase came mainly from shortening the interval between scheduled depot-level maintenance from 60 months to 48 months—thereby increasing the scheduled maintenance workload— and not from having to divert money to deal with unscheduled maintenance. For the deepwater cutters, the amount of estimated deferred maintenance increased from fiscal year 2002 to 2003, but then dropped significantly in fiscal year 2004. The decrease in fiscal year 2004 came mainly because (1) the Coast Guard ceased maintenance on an icebreaker, thus freeing up some maintenance funds; and (2) the Coast Guard also received additional operational and maintenance funding, allowing it to deal with both scheduled and unscheduled maintenance. Thus, the drop in the estimate of deferred maintenance costs for fiscal year 2004 is not necessarily an indicator that the condition of the legacy assets was improving; it could result from the Coast Guard having more money to address the maintenance needs.

Current Condition Measures Not Robust Enough to Clearly Link Condition with Effect on Mission Capabilities

At the time we began our work, the Coast Guard’s condition measures were not sufficiently robust to systematically link assets’ condition with degradation in mission capabilities. As we discussed with Coast Guard officials, without such condition measures, the extent and severity of the decline in the existing deepwater legacy assets and their true condition cannot be fully determined. As a result, the picture that emerges regarding the condition of the deepwater legacy assets based on current Coast Guard condition measures should be viewed with some caution. While there is no systematic, quantitative evidence sufficient to demonstrate that deepwater legacy assets its deepwater legacy assets are “failing at an unsustainable rate” as the Coast Guard has asserted, this does not mean the assets are in good condition or have been performing their missions safely, reliably and at levels that meet or exceed Coast Guard standards. We identified two factors that need to be considered to put these condition measures in proper context.

The first factor deals with limitations in the measures themselves. Simply put, the Coast Guard’s measures of asset condition do not fully capture the extent of the problems. As such, they may understate the decline in the legacy assets’ condition. More specifically, Coast Guard measures focus on events, such as flight mishaps or equipment casualties, but do not measure the extent to which these and other incidents degrade mission capabilities. Here are two examples in which the Coast Guard’s current measures are not sufficiently robust to systematically capture degradation in mission capabilities:

The surface search radar system on the HC-130 long-range surveillance aircraft, called the APS-137 radar, is subject to frequent failures and is quickly becoming unsupportable, according to Coast Guard staff with whom we met. Flight crews use this radar to search for vessels in trouble and to monitor ships for illegal activity, such as transporting illicit drugs or illegal immigrants. When the radar fails, flight crews are reduced to looking out the window for targets, greatly reducing mission efficiency and effectiveness. A flight crew in Kodiak, Alaska, described this situation as being “like trying to locate a boat looking through a straw.” Mission capability degradations such as these are not reflected in the Coast Guard’s current condition measures.

The 378-foot cutter Jarvis recently experienced a failure in one of its two main gas turbines shortly after embarking on a living marine resources and search and rescue mission. While Jarvis was able to accomplish its given mission, albeit at reduced speeds, this casualty rendered the cutter unable to respond to any emergency request it might have received—but did not in this case—to undertake a mission requiring higher speeds, such as drug interdiction. The Coast Guard condition measures are not robust enough to capture these distinctions in mission capability.

The second factor that needs to be kept in mind is the compelling nature of the other evidence we gathered apart from the Coast Guard’s condition measures. This evidence, gleaned from information collected during our site visits and discussions with maintenance personnel, indicated deteriorating and obsolete systems and equipment as a major cause of the reduction in mission capabilities for a number of deepwater legacy aircraft and cutters. Such problems, however, are not captured by the Coast Guard’s condition measures. One example of this involves the HH-65 short-range recovery helicopter. While this helicopter consistently exceeded availability standards established by the Coast Guard over the 5-year period we examined, it is currently operating with underpowered engines that have become increasingly subject to power failures. As a result, Coast Guard pilots employ a number of work arounds, such as dumping fuel or leaving the rescue swimmer on scene if the load becomes too heavy. Further, because of increasing safety and reliability problems, the Coast Guard has also implemented a number of operational restrictions—such as not allowing the helicopter to land on helipads—to safeguard crew and passengers and prevent mishaps until all of the fleet’s engines can be replaced.

The Coast Guard has recently recognized the need for improved measures to more accurately capture data on the extent to which its deepwater legacy assets are degraded in their mission capabilities, but as of April 2005, such measures had not yet been finalized or implemented. Subsequent to our inquiries regarding the lack of condition and mission capability measures, Coast Guard naval engineers reported that they had begun developing a “percent of time fully mission capable” measure to reflect the degree of mission capability, as well as measures to track cutter readiness. We agree that measures like this are needed—and as soon as possible. Further, current plans call for the measure, if approved, to be used for cutters, but not for aircraft. Consequently, even if this measure were to be implemented across the Coast Guard, there would still be no measure to address degradation in mission capabilities for aircraft. We will be exploring this issue further in our follow-on report.

Actions to Maintain, Upgrade, and Better Manage Deepwater Legacy Assets Are Under Way, but Condition Issues Remain

The Coast Guard has taken several actions to maintain, upgrade, and better manage its deepwater legacy assets. These include establishing a compendium of information for making decisions regarding maintenance and upgrades; performing more extensive maintenance between deployments; applying new business rules and strategies, at the Pacific Area Command, to better sustain the 378-foot high-endurance cutters through 2016; and exploring additional strategies for prioritizing the maintenance and capability enhancement projects needed on its legacy assets in an effort to provide more objective data on where to best spend budget dollars to achieve the greatest enhancements in mission capabilities. These additional efforts are likely helping to prevent a more rapid decline in the condition of these assets, but condition problems continue, and the efforts will likely involve additional costs.

Compendium of Needs Is Being Compiled and Used

Since 2002, the Coast Guard has annually issued a Systems Integrated Near Term Support Strategy compendium. Among other things, this compendium consolidates information needed to make planning and budgeting decisions regarding maintenance and upgrades to sustain legacy assets. Its purpose is to serve as a tool for senior Coast Guard management in setting priorities and planning budgets. From this strategic document, the Coast Guard has identified a number of upgrades to improve the capabilities of the deepwater legacy aircraft and cutters. The most recent compendium (for fiscal year 2006) lists more than $1 billion worth of upgrades to the deepwater legacy assets. The planned upgrades identified in the compendium that have been approved and received initial funding account for an estimated $856 million the Coast Guard anticipates it will need to complete those projects. The approved upgrades for deepwater legacy assets are shown in table 4.

Among the projects already begun is the re-engining of the HH-65 helicopters to increase the helicopter’s power and capabilities. The Coast Guard is also upgrading several other aviation systems in an effort to improve aircraft capabilities. Enhancements are also planned for certain classes of deepwater cutters. For example, during this fiscal year, the Coast Guard is to begin a maintenance effectiveness project on the 210 foot and 270-foot cutters. This project includes replacing major engineering subsystems with the goal of extending the cutters’ service lives until their replacement by the Offshore Patrol Cutter. Of the $856 million total estimated costs needed for the planned upgrades to the deepwater legacy assets listed above, $215 million has been allocated through fiscal year 2005 and the Coast Guard has requested another $217.3 million in its fiscal year 2006 budget. The remaining estimated costs of $423.7 million would have to be funded beyond fiscal year 2006.

Increasing Amounts of Maintenance Are Being Performed, but Loss of Mission Capabilities Continues

Coast Guard personnel consistently reported to us that crew members have to spend increasingly more time between missions to prepare for the next deployment. For example, to prevent further corrosion-related problems, air station maintenance personnel at the locations we visited said they have instituted additional measures, such as washing and applying fluid film to the aircraft prior to each deployment. Similar accounts were told by personnel working on cutters. For example, officers of the 270-foot cutter Northland told us that because of dated equipment and the deteriorating condition of its piping and other subsystems, crew members have to spend increasingly more time and resources while in port to prepare for their next deployment. While we could not verify these increases in time and resources because the Coast Guard does not capture data on these additional maintenance efforts, the need for increasing amounts of maintenance was a message we consistently heard from the operations and maintenance personnel with whom we met.

Such efforts are likely helping to prevent a more rapid decline in the condition of these deepwater legacy assets, but it is important to note that even with the increasing amounts of maintenance, these assets are still losing mission capabilities because of deteriorating equipment and system failures. For example, in fiscal year 2004, one 378-foot cutter lost 98 counterdrug mission days because of a number of patrol-ending casualties—including the loss of ability to raise and lower boats and run major electrical equipment—requiring $1.2 million in emergency maintenance. Another 378-foot cutter lost 27 counterdrug mission days in the fall of 2004 when it required emergency dry-dock maintenance because of hydraulic oil leaking into the reduction gear.

New Initiative for Maintaining 378-Foot Cutters Is Under Way

One effort is under way at the Coast Guard’s Pacific Area Command to improve maintenance practices for the 378-foot cutters. Pacific Area Command officials have recognized that a different approach to maintaining and sustaining legacy cutters may be needed and, as a first step, they have undertaken an initiative applying what they refer to as “new business rules and strategies” to better maintain the 378-foot high endurance cutters through 2016. Under the original Deepwater proposal, the final 378-foot cutter was to be decommissioned in 2013, but by 2005, that date had slipped to 2016. To help keep these cutters running through this date, Pacific Area Command officials are applying such rules and strategies as (1) ensuring that operations and maintenance staffs work closely together to determine priorities, (2) recognizing that maintaining or enhancing cutter capabilities will involve trade-off determinations, and (3) accepting the proposition that with limited funding not all cutters will be fully capable to perform all types of missions. Pacific Area Command officials believe that in combination, these principles and strategies will result in more cost-effective maintenance and resource allocation decisions—recognizing that difficult decisions will still have to be made to balance maintenance and operations.

The Pacific Area Command’s new initiative has the potential for assisting the Coast Guard in making more informed choices regarding the best use of their resources, but the approach will likely require that the Coast Guard allocate additional maintenance funds. In particular, the Pacific Area Commander told us that in order for the 378-foot cutters to be properly maintained until their replacements become operational; the Coast Guard will have to provide additional funding for sustaining the 378 foot cutters. So far, the Coast Guard’s budget plans or requests do not address this potential need.

Coast Guard is Developing a Strategy to Better Prioritize Upgrades and Maximize Asset Capabilities

In the past, we have recommended that the Coast Guard develop a long term strategy to set and assess levels of mission performance. We found this was an important step for the Coast Guard to take because it links mission performance levels to measurable outputs and goals so that the Coast Guard can better decide how limited budget dollars should be spent. The Coast Guard has recently begun to apply the principles behind such a strategy to (1) better prioritize the projects needed to upgrade legacy assets that will be part of the Deepwater program and (2) obtain the greatest overall mix of capabilities for its assets within its budget in order to maximize mission performance. The tool it is developing is called the Capital Asset Management Strategy (CAMS).

CAMS, once fully implemented, is expected to help the Coast Guard to better manage its assets by linking funding decisions to asset condition. Unlike the Coast Guard’s current compendium, CAMS is designed to provide analyses on the capability trade-offs for upgrades and maintenance projects across asset classes, thereby allowing the Coast Guard to determine which combination of projects will provide the most capability for the dollars invested. For example, when trying to decide among potential project upgrades such as a HC-130 weather radar replacement, an HH-65 sliding cabin door replacement, or a 110-foot patrol boat fin stabilizer replacement, CAMS, once fully implemented, could provide the program managers with a recommended mix of project upgrades that would achieve the greatest capability enhancements based on the available budget.

CAMS analyses are to be based on legacy asset condition and readiness data, asset retirement and replacement timelines, asset degradation estimates, project production rates, cost data, and mission utility rankings. Mission utility rankings will grade an asset’s importance to specific missions, such as search and rescue or counterdrug operations. Rankings may also be assigned to an asset’s critical subsystems, or may be altered based on an asset’s geographic location. For example, a 378-foot cutter may be critical to the success of fisheries patrols in the Pacific, but may not be as important for alien/migrant interdiction operations in the Caribbean. In addition, the Coast Guard plans to rank its missions based on their relative importance. Each of these elements is to form the basis for recommendations regarding which combination of upgrade and maintenance projects will provide the greatest enhancements to fleet capabilities.

CAMS recommendations are not intended to be a replacement for the budget development process, but rather are to augment the information currently provided to decision-makers and be reviewed by several internal Coast Guard officials before final funding decisions are made. Further, in order to prevent user “gaming”—making assumptions in such a way as to assure a positive recommendation or outcome for a particular project— the Coast Guard is developing a series of job aids, manuals and training courses to ensure data consistency.

Coast Guard officials expect to have the CAMS fully implemented by September 2005 and intend to use it while developing the Coast Guard’s fiscal year 2008 budget submission. Although it is too soon to assess the effectiveness of CAMS, we view this approach as a good faith effort toward knowledge-based budgeting for legacy asset sustainment.

Management Challenges Faced in Acquiring New Assets Remain Significant

Since the inception of the Deepwater program, we have expressed concerns about the degree of risk in the acquisition approach and the Coast Guard’s ability to manage and oversee the program. Last year, we reported that, well into the contract’s second year, key components needed to manage the program and oversee the system integrator’s performance had not been effectively implemented. We also reported that the degree to which the program was on track could not be determined, because the Coast Guard was not updating its schedule. We detailed needed improvements in a number of areas, shown in table 5. These concerns have a direct bearing on any consideration to increase the program’s pace. Because the Coast Guard was having difficulty managing the Deepwater program at the pace it had anticipated, increasing the pace by attempting to speed the acquisition would only complicate the problem.

The Coast Guard agreed with nearly all of our recommendations and has made progress in implementing some of them. In most cases, however, while actions are under way, management challenges remain that are likely to take some time to fully address.

Improvement of Program Management and Contractor Oversight Is Mixed

We have seen mixed success in the Coast Guard’s efforts to improve management of the program and contractor oversight. All four areas of concern—improving integrated project teams (IPT) , ensuring adequate staff for the program, planning for human capital requirements for field units receiving new assets, and updating the program’s schedule—have yet to be fully addressed.

Strengthening Integrated Product Teams

Although the Deepwater program has made some efforts to improve the effectiveness of IPTs, we continue to see evidence that more improvements are needed for the teams to do their jobs effectively. These teams, the Coast Guard’s primary tool for managing the program and overseeing the contractor, are generally chaired by a subcontractor representative and consist of members from subcontractors and the Coast Guard. The teams are responsible for overall program planning and management, asset integration, and overseeing delivery of specific Deepwater assets. Since our March 2004 report, the teams have been restructured, and 20 teams have charters setting forth their purpose, authority, and performance goals. And new, entry-level training is being provided to team members.

Despite this progress, however, the needed changes are not yet sufficiently in place. A recent assessment by the Coast Guard of the system integrator’s performance found that roles and responsibilities in some teams continue to be unclear. Decision making is to a large extent stovepiped, and some teams still lack adequate authority to make decisions within their realm of responsibility. One source of difficulty for some team members has been the fact that each of the two major subcontractors has used its own databases and processes to manage different segments of the program. Decisions on air assets are made by Lockheed Martin, while decisions regarding surface assets are made by Northrop Grumman. This approach can lessen the likelihood that a “system of systems” outcome will be achieved. Officials told us that more attention is being paid to taking a systemwide approach and that the Coast Guard has emphasized the need to ensure that the two major subcontractors integrate their management systems.

Ensuring Adequate Staffing for the Deepwater Program

The Coast Guard has taken steps to more fully staff the Deepwater program, with mixed effects. In February 2005, the Deepwater program executive officer approved a revised human capital plan. The plan emphasizes workforce planning, including determining needed knowledge, skills, and abilities and developing ways to leverage institutional knowledge as staff rotate out of the program. This analysis is intended to help determine what gaps exist between needed skills and existing skills and to develop a plan to bridge these gaps. The Coast Guard has also taken some short-term steps to improve Deepwater program staffing, hiring contractors to assist with program support functions, shifting some positions from military to civilian to mitigate turnover risk, and identifying hard-to-fill positions and developing recruitment plans specifically for them. Finally, the Deepwater program and the Coast Guard’s acquisition branch have begun using an automated system for forecasting military rotation cycles, a step Deepwater officials believe will help with long range strategic workforce planning and analysis.

Despite these actions, however, vacancies remain in the program, and some metrics that may have highlighted the need for more stability in the program’s staff have been removed from the new human capital plan. As of January 2005, 244 positions were assigned to the program, but only 206 of these were filled, a 16 percent vacancy rate. A year ago, 209 staff were assigned to the program. Further, the new human capital plan removes a performance goal that measured the percentage of billets filled at any given time. Coast Guard officials acknowledged that the prior plan’s goal of a 95 percent or higher fill rate was unduly optimistic and was a poor measure of the Coast Guard’s ability to meet its hiring goals. For example, billets for military personnel who plan to rotate into the program in the summer are created at the beginning of the budget year, leading the metric to count those positions as vacant from the beginning of the budget year until summer. Other performance metrics that were included in the prior plan to measure progress in human capital issues have also been removed. For example, to help ensure that incoming personnel received acquisition training and on-the-job training, a billet was included in the prior plan to serve as a floating training position that replacement personnel could use for a year before the departure of military incumbents. This position was never funded, and the new plan removes the billet.

Improving Communication with Personnel Who Will Use the New Assets

The Coast Guard recognizes the critical need to inform the operators who are to use the Deepwater assets of progress in the program, and officials stated that, on the basis of our recommendations, they have made a number of improvements in this area. A November 2004 analysis of the Deepwater program’s communication process, conducted in coordination with the National Graduate School, found that the communication and feedback processes were inadequate. Emphasis has now been placed on outreach to field personnel, with a multipronged approach involving customer surveys, face-to-face meetings, and presentations. We have not yet evaluated the effectiveness of the new approach.

Human capital requirements for the Deepwater program—such as crew numbers and schedules, training, and support personnel—will have an increasing impact on the program’s ability to meet its goals as the pace at which assets are delivered to field units picks up. Recent assessments by Coast Guard performance monitors show this to be an area of concern.Coast Guard officials have expressed concern about whether the system integrator is appropriately considering human capital in systems engineering decisions. The system integrator is required to develop a workforce management plan for Deepwater, as well as “human factors engineering” plans for each Deepwater asset and for the overall system of systems. The Coast Guard rejected the contractor’s workforce management plan and several of the proposed human factors engineering plans as being inadequate. The rejections were due, in part, to the lack of an established and integrated system-level engineering approach that shows how issues relating to human capabilities and limitations of actually performing with the system will be approached. One performance monitor noted that, as of late 2004, requirements for staffing and training of maintenance facilities and organizations had yet to be determined. According to the Coast Guard, emphasis on a contractor’s approach to addressing human capital considerations is necessary to ensure that Deepwater goals are met, especially as they pertain to operational effectiveness and total ownership cost.

Updating the Acquisition Schedule

The Coast Guard has recently undertaken efforts to update the original 2002 Deepwater acquisition schedule—an action that we suggested in our June 2004 report. The original schedule had milestone dates showing when work on an asset would begin and when delivery would be expected, as well as the integrated schedules of critical linkages between assets, but we found that the Coast Guard was not maintaining an updated and integrated version of the schedule. As a result, the Coast Guard could not demonstrate whether individual components and assets were being integrated and delivered on schedule and in critical sequence. As recently as October 2004, Deepwater performance monitors likewise expressed concern that the Coast Guard lacked adequate visibility into the program’s status and that lack of visibility into the schedules for component-level items prevented reliable forecasting and risk analysis. The Coast Guard has since taken steps to update the outdated schedule and has indicated that it plans to continue to update the schedule each month for internal management purposes and semiannually to support its budget planning efforts. We think this is an important step toward improving the Coast Guard’s management of the program because it provides a more tangible picture of progress as well as a baseline for holding contractors accountable. We will continue to work closely with the Coast Guard to ensure progress is made and to monitor how risks are mitigated.

Procedures for Ensuring System Integrator Accountability Are More Rigorous, but Concerns Remain

We have seen progress in terms of the rigor with which the Coast Guard is periodically assessing the system integrator’s performance, but concerns remain about the broader issues of accountability for achieving the overarching goals of minimizing total ownership costs and maximizing operational effectiveness.

Improving Criteria for Assessing Performance

Improvements continue to be made to the criteria for assessing the system integrator’s performance. In March 2004, we reported that the process for assessing performance against specific contract tasks lacked rigor. The criteria for doing so have since been revised to more clearly reflect those that are objective, (that is, measured through automated tools against established metrics), and those that are subjective, meaning the narrative comments by Coast Guard performance monitors. Weights have been assigned to each set of evaluation factors, and the Coast Guard continues to refine the distribution of the weights to reach an appropriate balance between automated results and the eyewitness observations of the performance monitors. Coast Guard officials told us that they have also provided additional guidance and training to performance monitors. We found that efforts have been made to improve the consistency of the format used for their input in assessments of the system integrator’s performance. Coast Guard officials said that they are continuing to make improvements to ensure that performance monitors’ relevant observations are appropriately considered in making award fee determinations.

It is important to note that although performance monitor comments are considered subjective, they are valuable inputs to assessing the system integrator’s performance, particularly when they are tied to measurable outcomes. It will be necessary for the Coast Guard to continue refining the award fee factors as the program progresses. In some cases, we noted that the performance monitors’ assessments differed vastly from the results of automated, data-driven assessments. For example, while schedule management is discussed in the Coast Guard’s most recent assessment of the system integrator’s performance as a major area of challenge and risk, the objective measure showed 100 percent compliance in this area. Another metric assesses the extent to which integrated product teams consider the impact of their decisions on the overall cost and effectiveness of the Deepwater program. Performance monitors reported that because system-level guidance had not been provided to the teams responsible for specific assets, they had a limited ability to see the whole picture and understand the impact of decisions on total ownership costs and operational effectiveness. However, the automated measure was again 100 percent compliance. Coast Guard officials said that, in some cases, the data-driven metrics do not accurately reflect the contractor’s performance. For the next award fee assessment, Deepwater officials plan to revise the metrics and place more weight on the performance monitors’ input, while ensuring that it is based on measurable outcomes.

Holding the System Integrator Accountable for Effectiveness of Project Teams

Changes have been made to the award fee metrics that place additional emphasis on the system integrator’s responsibility for making integrated project teams effective. Award fee criteria now incorporate specific aspects of how the integrator is managing the program, including administration, management commitment, collaboration, training, and empowerment of these teams. However, as discussed above, concerns remain about whether the teams are effectively accomplishing their goals.

Evaluation of Operational Effectiveness and Total Ownership Cost

While the Coast Guard has developed models to measure the system integrator’s performance in operational effectiveness and total ownership costs, concrete results have not yet emerged. Minimizing total ownership costs and maximizing operational effectiveness are two of the overarching goals of the Deepwater program. The system integrator’s performance in these two areas will be a critical piece of information when the Coast Guard makes a decision about whether to award the contractor the first contract option period of 5 years. Initial decision making is to start next year.

With regard to the operational effectiveness of the program, measuring the system integrator’s impact has yielded limited results to date because few of the new assets are operational. The Coast Guard has developed modeling capabilities to simulate the effect of the new capabilities on its ability to meet its missions. However, until additional assets become operational, progress toward this goal will be difficult to determine.

With regard to total ownership costs, the Coast Guard does not plan to implement our recommendation. It has not adhered to its original plan, set forth in the Deepwater program management plan, of establishing as its baseline a cost not to exceed the dollar value of replacing the assets under a traditional approach (e.g., on an asset-by-asset basis rather than a system-of-systems approach). Although a cost baseline consistent with the program management plan’s approach was initially established, this number has not been rebaselined, as has the system integrator’s cost estimate baseline, and is not being used to evaluate the contractor’s progress in holding total ownership costs down. In practice, the baseline being used to measure total ownership cost is the system integrator’s own cost estimate. As we reported in March 2004, we believe that measuring the system integrator’s cost growth compared with its own cost proposal will tell the government nothing about whether it is gaining efficiencies by turning to the system of systems concept.

Establishing Criteria and Documenting Changes to the Baseline

Coast Guard officials stated that the contract total ownership cost and operational effectiveness baseline is adjusted based on approved decision memorandums from the Agency Acquisition Executive, the Vice Commandant of the Coast Guard.

Coast Guard Has Taken Steps to Hold the System Integrator Accountable for Competition

The Coast Guard reported taking steps to address our recommendations concerning cost control through competition. Our recommendations pertained to competition among second-tier suppliers and notification of “make” decisions.

Competition among second-tier suppliers. Coast Guard officials told us that in making the decision about whether to award the first contract option, the government will specifically examine the system integrator’s ability to control costs by assessing the degree to which competition is fostered at the major subcontractor level. The evaluation will consider the subcontractors’ project management structure and processes to control costs, as well as how market surveys of similar assets and major subsystems are implemented. The Coast Guard is focusing its attention on those areas that were priced after the initial competition for the Deepwater contract was completed, such as the HH-65 re-engining and the C-130J missionization. For example, a new process implemented for the C-130J missionization was a requirement for competition in subcontracting and government approval of all subcontracts exceeding $2 million in order for the Coast Guard to monitor the integrator’s competition efforts.

Notification of make decisions. According to the Federal Acquisition Regulation, the prime contractor is responsible for managing contract performance, including planning, placing, and administering subcontracts as necessary to ensure the lowest overall cost and technical risk to the government. When “make-or-buy programs” are required, the government may reserve the right to review and agree on the contractor’s make-or-buy program when necessary to ensure negotiation of reasonable contract prices, among other things. We recommended that the Coast Guard be notified of make decisions over $5 million in order to facilitate controlling costs through competition. We suggested the $5 million threshold because Lockheed Martin, one of the major subcontractors, considers that amount to be the threshold for considering its suppliers major. The Coast Guard has asked the system integrator, on a voluntary basis, to provide notification one week in advance of a make decision of $10 million or more based on the criteria in the Federal Acquisition Regulation.According to Coast Guard officials, to date, no make decision has exceeded $10 million since the request was made. The details implementing this recommendation have not yet been worked out, such as specifically who in the Coast Guard will monitor the subcontractors’ make decisions to ensure that the voluntary agreement is complied with.

Concluding Observations

Our work to date suggests the costly and important Deepwater program will need constant monitoring and management attention to successfully accomplish its goals. In this respect, we identified three points that should be kept in mind in considering how to proceed with the program.

First, the need to replace or upgrade deteriorating legacy assets is considerable. While the Coast Guard is making progress on developing (1) measures that better demonstrate how the deteriorating condition of the legacy assets impact on mission capabilities and (2) a strategy to better prioritize upgrades and maximize capabilities, it is clear that the deepwater legacy assets are insufficient for meeting all of the Coast Guard’s missions.

Second, although the need to replace and upgrade assets is strong, there still are major risks in the Coast Guard’s acquisition approach. The cost increases and schedule slippages that have already occurred are warning signs. We will continue to work with the Coast Guard to determine how best to manage these risks so that the Deepwater missions can be accomplished in the most cost-effective way.

Third, there are signs that as the Deepwater program moves ahead, the Coast Guard will continue to report more problems with sustaining existing assets, together with the attendant need for additional infusions of funding to deal with them. Some of these problems, such as those on the 378-foot cutters, are included in the compendium the Coast Guard uses to set sustainment priorities and plan budgets, but the Coast Guard has not allocated funds because the problems pertain to assets that are among the first to be replaced. However, projects to address these problems are nevertheless likely to be needed. We will continue to work with the Coast Guard to determine if there is a more systematic and comprehensive approach to keeping the Congress abreast of the potential bill for sustaining these assets.

Mr. Chairman and Members of the Subcommittee, this completes my prepared statement. I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time.

Contacts and Acknowledgements

For information about this testimony, please contact Margaret Wrightson, Director, Homeland Security and Justice Issues, at (415) 904-2200, or wrightsonm@gao.gov. Other individuals making key contributions to this testimony include Steven Calvo, Jerry Clark, Christopher Conrad, Adam Couvillion, Michele Fejfar, Geoffrey Hamilton, Julie Leetch, Michele Mackin, Christopher Miller, Stan Stenersen, and Linda Kay Willard.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.