Coast Guard Acquisitions: Enhanced Oversight of Testing Could Benefit National Security Cutter Program and Future DHS Acquisitions
Released: 2016-02-03T00:00:00

Letter

Background

DHS and Coast Guard acquisition guidelines require operational test and evaluation by an independent test agency to confirm that the production configured system meet all requirements before approval for full-rate production. The Coast Guard uses the U.S. Navy’s COTF to conduct operational tests and other evaluations for its major acquisition programs according to those programs’ requirements. COTF serves as an independent evaluator of an asset’s capabilities and has experience testing U.S. Navy assets.

In conducting operational testing, COTF evaluates an asset’s operational effectiveness and suitability:

For operational effectiveness, testers determine whether or not an asset can meet its missions.

For operational suitability, testers determine whether or not the agency can logistically support the asset to an acceptable standard, such as having the asset available for operations 85 percent of its scheduled deployment time.

Critical operational issues (COI) are one metric used to determine an asset’s operational effectiveness and suitability and are stated in the form of a question. COIs are examined during testing to evaluate a system’s ability to provide the desired capability and perform its mission. COTF assessed the NSC’s COIs, for example, by comparing the outcome of the test event against the full scope of the COI to determine whether the COI was met or not. Unmet COIs are often the result of related deficiencies, which are identified during testing and include any system that is lacking in its ability to meet normal standards or to function as intended. Deficiencies are scored based on the severity of the problem and its impact on the asset’s ability to accomplish its mission. COIs and deficiencies identified during testing both factor into an asset’s overall operational effectiveness and suitability rating.

In addition to verifying that an asset is operationally effective and suitable, operational testing also tests key performance parameters (KPP), which are the capabilities considered essential for mission accomplishment. KPPs are listed by threshold values, which are the minimum acceptable level of performance, and objective values, which are the desired level of performance. For example, a KPP for the NSC is being able to reach a maximum speed of 28 knots for a threshold value and 31 knots for an objective value. KPPs differ from COIs in that KPPs focus on specific performance metrics, while COIs focus on certain types of missions that an asset should be able to conduct or an asset’s ability to be ready to perform those missions. Table 1 provides examples of COIs and KPPs for the NSC.

Operational testing can occur over many test events. Two of those key test events are: Initial Operational Test and Evaluation (IOT&E): This event is meant to gather the data necessary to resolve COIs, determine an asset’s operational effectiveness and suitability, and, according to Coast Guard acquisition guidance, occur prior to a full-rate production decision. The test event concludes with a rating of operationally effective or not effective, operationally suitable or not suitable.

Follow-on operational test and evaluation (FOT&E): This event is conducted after IOT&E and an asset’s full rate production decision and focuses on refining the conclusions that were made during previous operational test events, evaluating production changes, and reevaluating the system to ensure that it continues to meet operational needs. It also validates any incomplete or deferred requirements and verifies the correction of deficiencies identified during IOT&E. FOT&E concludes with an operational effectiveness and suitability rating similar to that of IOT&E.

Following IOT&E and FOT&E, COTF writes a test report that focuses on the resolution of the asset’s COIs and any deficiencies that were identified during testing. These reports typically include a summary of the resolution of the asset’s COIs.

While Initial Operational Testing Revealed Some Major Deficiencies, the NSC Met Most of Its Key Performance Parameters

As we reported in January 2016, IOT&E took place about 2 years later than planned and after 7 of the 8 planned NSCs were under contract, with 3 operational. We have previously found that delaying critical test events can lead to late discoveries and could result in additional design changes and costs to programs. The 8 NSCs are planned to be fully operational by 2020 and the Coast Guard is phasing out the legacy 378-foot High Endurance Cutters as the NSCs enter operations. During testing, the NSC successfully demonstrated 18 of its 19 COIs, with one COI— cybersecurity—being deferred to FOT&E. This deferral was due to the DHS Director of Operational Test and Evaluation (DOT&E) postponing the testing of the NSC’s cybersecurity capabilities until a more robust test plan could be developed to reflect emerging threats.

At the conclusion of IOT&E, COTF found the NSC to be operationally effective and suitable, but with 10 major deficiencies. None of the major deficiencies were rated as severe, which would preclude the NSC from accomplishing its mission. Five of the 10 major deficiencies pertained to the NSC’s weapon systems and cutter boats. Table 2 shows the 10 deficiencies.

In its assessment of the NSC’s IOT&E event, DOT&E stated that the reliability and operational availability issues of the weapon systems—the CIWS, NULKA Launcher, TRS-3D air search radar, and the 57-mm gun— affect the overall ability of the NSC to conduct certain missions. While the CIWS, NULKA launcher, and air search radar were all repaired following IOT&E, post-operational reports indicate that problems persist with these systems as they were often unavailable during operations. Despite these findings, as noted above, COTF found the NSC to be operationally effective and suitable.

While COIs and deficiencies factor into a system’s operational effectiveness and suitability rating, KPPs are measures of the capabilities considered essential to mission accomplishment. In our January 2016 report, we found that during IOT&E and other test events, the NSC fully met 12 of its 19 KPPs. However, by not meeting all KPPs, the Coast Guard is not able to demonstrate that the NSC is providing the capabilities that it intended to field. For instance, the Coast Guard has not yet demonstrated that the NSC can achieve a hard and soft kill against a subsonic cruise missile as required, or fully meet interoperability requirements with the Department of Defense, DHS, and other government agencies. Table 3 displays the 7 KPPs not fully met for the NSC, the test results, and a discussion of these results.

Of the 7 KPPs not met, 3 pertain to the NSC’s cutter boat operations. The cutter boats are designed to be integral to the NSC’s overall capability, operate both within and beyond the visual range of the NSC, and enhance the overall mission effectiveness of the NSC in every mandated mission area. We found that the Coast Guard and COTF have different interpretations of the cutter boat KPPs. COTF conducted IOT&E according to the NSC’s test and evaluation plan and determined that the three KPPs involving cutter boats were not fully met since the boats are unable to operate through all sea state 5 conditions. According to Coast Guard officials, the September 2012 requirements document for the NSC should have been written more clearly to convey the sea state expectation for cutter boat launch and recovery operations, since the NSC’s sea state KPP was never intended to be applicable to the operation of the cutter boats. In January 2016 we recommended, and the Coast Guard concurred, that the NSC’s KPPs for the operation of the cutter boats should be clarified.

Unclear Guidance on Follow-on Testing May Lead to NSCs and Future DHS Assets Deploying without Having Demonstrated Full Capabilities

As we reported in January 2016, according to COTF officials, FOT&E will begin in the fall of 2016 and is scheduled to continue through at least 2017. Following IOT&E, DHS held an acquisition review board (ARB) to discuss the outcome of IOT&E, which resulted in DHS approving the NSC program for full rate production in October 2014. ARBs review major acquisition programs for proper management, oversight, accountability, and alignment with DHS’s strategic functions at acquisition decision events and other meetings as needed. The resulting acquisition decision memorandum (ADM) from October 2014 directed the Coast Guard to conduct FOT&E and complete three action items: (1) complete testing of the cybersecurity COI; (2) verify the correction of all major deficiencies, including the unmet KPPs; and (3) assess the NSC’s cyber-security capabilities. The cybersecurity COI is planned to be tested in 2016, which, if successful, will address the first and third requirements of the ADM, but other testing events are expected to occur through 2017 and possibly beyond.

The ADM also directed the Coast Guard to verify the correction of all deficiencies, including the 7 unmet KPPs. According to Coast Guard officials, they have corrected 4 of the 10 major deficiencies from IOT&E that involved equipment failures by restoring the operational status of the related systems on the Stratton, and they have plans to correct four more. However, according to Coast Guard documentation and officials, they may not correct 2 deficiencies due to the cost of making fleet-wide changes, and because the Coast Guard has developed an interim solution. Table 4 shows the Coast Guard’s plans, as we reported in January 2016, for resolving the major deficiencies.

As we also found in January 2016, while the Coast Guard has plans to conduct FOT&E for the NSC, it will have accepted the delivery of at least the 6th NSC before the testing is complete, meaning that the Coast Guard will be operating 6 NSCs before it has resolved issues from IOT&E and knows the cutter’s full capabilities. DHS’s guidance for its major acquisitions does not require programs to conduct FOT&E, nor does it specify the timing of FOT&E or the actions that should be taken following the completion of testing. Further, DHS’s directive on test and evaluation does not include any direction or guidance on FOT&E.

We concluded in our January 2016 report that this gap in DHS guidance also has implications for future DHS assets. Most significantly, the Coast Guard is in the process of designing the OPC, which is the last of the major cutter classes to be built as part of the recapitalization program. This cutter class, which is intended to bridge the mission gap between the FRC and NSC, is estimated to cost $12.1 billion, making it the most expensive Coast Guard recapitalization program to date. Without updated guidance that establishes timeframes and responsibilities for completing all testing, the Coast Guard risks encountering the same scenario with the OPC—and other future DHS assets—that it has experienced with the NSC. That is, the Coast Guard could continue to buy assets without having demonstrated their full capabilities in testing. In January 2016 we recommended that updated guidance should establish factors that should be considered with planning FOT&E, including when test events will be concluded. We also recommended that an ARB be held, if necessary, to provide oversight and specify any further actions programs should take following FOT&E. DHS agreed with these recommendations and estimated they would be implemented by November 2016.

Problems Discovered Outside of IOT&E Are Preventing the Coast Guard from Operating Fully Capable NSCs

As we reported in January 2016, by the time of the spring 2014 IOT&E event, the Coast Guard had nearly four years of experience operating the NSCs. The Coast Guard has encountered several issues that require major retrofits and design changes on the NSC to correct problems encountered during operations and discovered during test events outside of IOT&E. The total cost of changes we identified as of June 2015 totals approximately $202 million. In order to minimize cost increases for some of these changes, the Coast Guard plans to maintain the original equipment design for the production of the remaining NSCs and plans to conduct retrofits after accepting delivery of the cutters. In some instances, replacement equipment is still in the prototype phase. The problems identified with these systems during operations will continue to impact the NSC until the design changes are implemented across the fleet. Figure 1 shows selected systems that will require retrofits after all eight cutters are built.

Early testing can allow performance issues to be discovered at a point when fixes can be incorporated into the design of an asset while it is still in production. As we have previously found for Department of Defense programs, continuing with full-rate production before ensuring that assets meet key requirements risks replicating problems in each new asset until such problems are corrected. The Coast Guard conducted IOT&E several years after it began operating the NSC and after the contracts for the majority of the fleet had been initiated. As a result, the Coast Guard plans to purchase and install equipment with known design flaws on the NSCs that are currently in production. Thus, the Coast Guard will be faced with paying for the replacement of these systems with new equipment that it must also purchase.

Furthermore, we found in January 2016 that the Coast Guard has encountered a variety of problems with the cutter’s propulsion systems during operations and, although there are several factors known to influence these problems, the root causes and the method and cost of potential solutions are not yet known. The problems include: (1) high engine temperatures, which limit the top speed of the cutter in certain conditions, (2) cracked cylinder heads, which are occurring at a rate higher than expected and are the NSCs number one operational degrader and cost driver for maintenance, and (3) overheating generator bearings, which have caused at least one patrol to be cut short due to the lack of an effective backup generator. Although the Coast Guard has two studies underway to identify the root causes of these problems, until the causes are identified and corrective actions implemented, the Coast Guard is at risk of experiencing costly and potentially mission-limiting problems with this equipment across the fleet. Thus, we recommended, and DHS agreed, to provide oversight and specify any further actions the NSC program should take at the conclusion of the studies related to the propulsion systems.

Observations on the FRC and OPC Acquisitions

As the Coast Guard has progressed in its acquisition of cutters, it has matured its acquisition processes, which has been demonstrated in its approach with the FRC and OPC programs. The process to date reflects some lessons learned from the NSC acquisition, for example in the areas of competition and the schedule for IOT&E. Furthermore, as the $12 billion OPC program moves forward, it may have opportunities to further incorporate some best practices that we have highlighted in our past work on shipbuilding.

As we reported in June 2014 and April 2015, the Coast Guard purchased the technical specifications and licenses necessary to build the FRC in order to use the information to conduct a full and open competition for the remaining 26 of 58 planned vessels. According to Coast Guard officials, the second phase of the acquisition was intended to promote competition and allowed bidders on the contract to make certain design changes to the ship, though the key performance parameters remain the same and the design for several critical systems—such as the propulsion system, generators, hull structure, and bridge layout—remain the same. The Coast Guard plans to award a contract for the second phase of this acquisition by the end of June 2016. We noted in June 2014 that when the government owns technical data rights, it does not need to rely on only one contractor to meet requirements.

As we also reported in April 2015, the Coast Guard is using a two- phased, competitive strategy to select a contactor to construct the OPC. In general, as we have previously found, competition is likely to save taxpayer dollars as opposed to a sole source acquisition approach, such as was used for the NSC. During the first phase for the OPC, the Coast Guard conducted a full and open competition to select three contractors to perform preliminary and contract design work, and, in February 2014, awarded firm-fixed price contracts to three shipbuilders. For the second phase, the Coast Guard plans to award, by the end of fiscal year 2016, a contract to one of these shipbuilders to complete the detailed design of the vessel and construct the first 9 to 11 ships. As we also reported, the Coast Guard plans to recompete the contract for the remaining vessels.

Competitive contracts can allow for the best return on investment for taxpayers by saving taxpayer money, conserving scarce resources, improving contractor performance, curbing fraud, and promoting accountability for results. According to Coast Guard officials, the Coast Guard currently plans to award the construction contract for the lead OPC ship in fiscal year 2018 and deliver this ship in 2021. As we found in April 2015, the OPC’s initial and full operational capability dates both slipped 15 months, which the Coast Guard attributes to procurement delays, including a bid protest that GAO ultimately denied in June 2014. The Coast Guard’s fiscal year 2016 Capital Investment Plan reflects $1.5 billion in funding for the OPC, which funds the detailed design work and construction of the first three vessels. After the first 3 of the planned fleet of 25 OPCs are built, the Coast Guard plans to increase its purchase to 2 OPCs per year until the final asset is delivered, currently scheduled for fiscal year 2035.

Regarding the timeframes for IOT&E, as we reported in January 2016, this event occurred for the NSC after 3 of its 8 planned cutters were operational. For the FRC, IOT&E occurred after 4 of the planned 58 cutters were operational. For the OPC, the Coast Guard plans to begin IOT&E by December 2023, by which time it expects to have 1 operational OPC of a planned buy of 25.

In addition to its efforts to enhance competition, the Coast Guard has developed a warranty provision under its contract with Bollinger Shipyards for the FRC program that has held the contractor responsible for production deficiencies. As we reported in June 2014, the Coast Guard does not always have insight into how much it costs the contractors to fix these issues. However, after multiple deficiencies interrupted production, officials noted they are confident that the Coast Guard has received value from this warranty. The Coast Guard plans to use these strategies when purchasing the OPC.

Based on best practices that we have previously identified, the Coast Guard may have opportunities to incorporate additional shipbuilding best practices with the OPC program. In May 2009, we reported on best practices that commercial shipbuilders use to ensure that ships are delivered on time and within budget. We found that before a contract is signed, a full understanding of the effort needed to design and construct the ship is reached, enabling commercial buyers and shipbuilders to sign a contract that fixes the price, delivery date, and ship performance parameters. To minimize risk, buyers and shipbuilders reuse previous designs to the extent possible and attain an in-depth understanding of new technologies included in the ship design. Before construction begins, commercial shipbuilders complete key design phases that correspond with the completion of a three-dimensional product model. Final information on the systems that will be installed on the ship is needed to allow design work to proceed. During construction, buyers maintain a presence in the shipyard and with key suppliers to ensure the ship meets quality expectations and is delivered on schedule. We will continue to assess the progress of the FRC and OPC acquisitions going forward.

Chairman Hunter, Ranking Member Garamendi, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions.

GAO Contact and Staff Acknowledgments

If you or your staff have any questions about this statement, please contact Michele Mackin at (202) 512-4841 or mackinm@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Katherine Trimble, Assistant Director; John Crawford; Lindsey Cross; and Peter W. Anderson.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.