Coast Guard: Progress Being Made on Deepwater Project, but Risks Remain
Released: 2001-05-02T00:00:00

Letter

Background

Cutters, patrol boats, airplanes, and helicopters are all critical to meeting the Coast Guard’s deepwater missions that are beyond the range of shore- based small boats. These missions include actions such as enforcing fisheries laws, intercepting drug smugglers and illegal immigrants, and conducting search and rescue missions far out at sea.

Why Does the Coast Guard Need to Replace or Modernize Deepwater Ships and Aircraft?

Many of the Coast Guard’s current cutters were built in the 1960s, and many of the aircraft in the 1970s and 1980s. Although these ships and aircraft have been upgraded in a number of ways since being acquired, the Coast Guard has documented a number of performance and support problems, such as the following: poor sensors and night operations capability on both aircraft and cutters, limited ability of cutters and aircraft to operate effectively together, inadequate communications, and high operating and maintenance costs.

In a November 1995 mission analysis report, the Coast Guard cited its rapidly aging deepwater fleet as a justification to begin a project for acquiring new ships and aircraft. In 1998, we reported that the service life of the Coast Guard’s deepwater ships and aircraft might be much longer than the Coast Guard originally estimated in its 1995 analysis. We recommended that the Coast Guard develop additional information on the remaining service life of ships and aircraft. In 1998, the Coast Guard determined that the service life of the various aircraft classes could be extended by about 11 to 28 years over original estimates, assuming that increased maintenance and upgrades occur. In addition, by January 2001, the Coast Guard had issued an updated analysis that extended the service life of two of the four ship classes by an additional 5 years, assuming that increased maintenance and upgrades occur. The Coast Guard provided this information to its contractors so that they could use it in developing their proposals.

In December 1999, an interagency task force on the roles and missions of the Coast Guard reported that recapitalization of the Coast Guard’s deepwater capability is a near-term national priority and endorsed the Deepwater Project’s process and timeline. Although our earlier work took issue with the Coast Guard’s initial analysis of how soon its deepwater assets would need to be replaced, we do not now take issue with the Coast Guard’s position that it needs to modernize these assets, especially due to the additional studies completed since our 1998 report. Our congressional directive in this report has been to examine the acquisition approach.

The Acquisition Approach for the Deepwater Project

The acquisition approach for the Deepwater Project is innovative. Rather than using the traditional approach of replacing an individual class of ships or aircraft, the Coast Guard has adopted a “system-of-systems” approach intended to integrate ships, aircraft, sensors, and communication links together as a system to accomplish mission objectives more effectively. The Coast Guard expects this approach will both improve the effectiveness of deepwater operations and reduce operating costs.

The project has two basic phases—a design phase (called “concept exploration” and known as phase 1) and a final proposal preparation and procurement phase (called “demonstration and validation/full-scale development” and known as phase 2). Phase 1 began in March 1998. As part of this phase, the Coast Guard contracted with three competing teams of contractors to conceive and begin designing a proposed deepwater system. Each proposal is to be based on meeting a set of performance specifications developed by the Coast Guard. Each team was instructed to develop its proposal on a funding stream of $300 million for the first year and $500 million annually until the project is completed. These amounts are in constant 1998 dollars; actual funding would be higher to account for inflation. Phase 1 ends with each team’s development of a proposed deepwater concept, the functional design for which will be 80-percent complete.

In phase 2, which begins in June 2001, the Coast Guard plans to issue a request for proposals (RFP) to the three industry teams to develop final proposals. The current schedule calls for these proposals to be completed and submitted to the Coast Guard during the last quarter of fiscal year 2001. The Coast Guard will evaluate which proposal provides best value for the government as gauged mainly by a combination of improvements in operational effectiveness and minimizing total ownership costs. Other evaluation factors include the technical feasibility of the proposed design and the management capability of the systems integrator.

When the deepwater contract is awarded early in 2002, the contract will actually be between the Coast Guard and the prime contractor, known as the “systems integrator,” of the winning contracting team. This systems integrator will be responsible for ensuring that each ship, aircraft, or other equipment is delivered on time and in accordance with agreed to prices. The systems integrator will also be called on to deliver the complete deepwater system in compliance with the Coast Guard’s system performance specifications. The Coast Guard adopted this approach because it does not believe it has the technical expertise or the resources to be a systems integrator. Also, the Coast Guard believes that a team of contractors led by a systems integrator would provide the best method of acquiring a set of ships, aircraft, and other equipment and would optimize improvements in operational effectiveness and total ownership costs. This contracting approach could thus result in a long-term contractual arrangement with a single contractor and its team of subcontractors.

The Coast Guard plans to have an initial 5-year contract with the systems integrator. The systems integrator would receive a base award for management and system integration services. Assuming the project proceeds as planned, task and delivery orders for deepwater equipment would be issued by the Coast Guard in accordance with the systems integrator’s implementation schedule. If the performance of the systems integrator is satisfactory for each award-term contract, the Coast Guard plans to award follow-on, award-term contracts (as many as five for successive 5-year, award-term contracts) with the same systems integrator. The Coast Guard plans to negotiate prices with the systems integrator on the follow-on contracts.

Viability of Contracting Approach Depends on a Sustained High Level of Funding

The Congress is at a critical juncture with the Deepwater Project because the success of the contracting approach rests heavily on the Coast Guard being able to count on sustained funding of about $500 million (in 1998 dollars) for 20 years or more. The contracting approach the Coast Guard has selected is not easily adaptable to lower levels of funding without stretching the schedule and increasing costs. However, there are signs that funding levels may be lower than the planned amount. Although the administration’s budget request for the Deepwater Project for fiscal year 2002 will be about 10 percent less than the project’s planned first-year funding, the average shortfall for fiscal years 2003 to 2006 is about 20 percent. Moreover, much of the funding for fiscal year 2002 is from the Western Hemisphere Drug Elimination Act (P.L. 105-277), a source that will not be available after this year unless the act is extended.

Capital funding for the Coast Guard is in competition with many other potential uses of federal funds within the Coast Guard itself, DOT as a whole, and other federal agencies. To accommodate the Deepwater Project, the Coast Guard is proposing to limit spending on its other ongoing capital projects to levels far below where they have been in decades. Given these various budgetary pressures, it appears advisable to have contractors develop their plans around a lesser amount. The Coast Guard’s approach, however, is inextricably tied to the more optimistic option. In using a lower, more realistic funding level aligned with OMB budget projections, the Coast Guard could lessen the risk of future cost increases, schedule stretch-outs, and low system performance levels.

Success of Contracting Approach Relies on Sustained High Funding

The contract approach that the Coast Guard has decided to use for the Deepwater Project depends on a large, sustained, and stable funding stream over the next 2 to 3 decades. The approach is based on acquiring ships and aircraft on the contractor’s proposed schedule so that they will form a “system of systems.” Substantial funding shortfalls cannot only affect the ships and aircraft scheduled for acquisition in the short term; but, they can also set off ripples affecting the acquisition of deepwater equipment for years to come. Adjustments that may be needed include revising the implementation plan for delivering equipment, renegotiating prices for deepwater equipment, and negotiating new cost and performance baselines with the systems integrator. Such adjustments would not only be costly; but, they could also slow the schedule to the point that (1) total ownership costs would rise and (2) advantages projected in the contractor’s proposal, such as improvements in operational effectiveness, would not materialize. At the extreme, funding shortfalls would affect the Coast Guard’s ability to proceed with the contract as well as the agency’s ability to perform its deepwater missions.

Administration’s Funding Proposals Signal Caution About Whether Projected Funding Will Be Available

The decision on funding the Deepwater Project rests ultimately with the Congress, but because this decision has yet to be made, we used the administration’s budget proposal for the Coast Guard (as contained in the budget documents prepared by OMB) as a starting point for analyzing the funding issue. OMB’s budget targets for fiscal years 2002 through 2006 do not propose specific amounts for the Deepwater Project; rather, they provide a single amount for all Coast Guard capital projects. As table 1 shows, this overall total ranges from $659 million (in-year-of expenditure dollars) in fiscal year 2002 to $719 million in fiscal year 2006. Because the Coast Guard has many other capital projects under way besides the Deepwater Project, it must decide how this money will be allocated among them. After receiving the budget targets from OMB in early March 2001, the Coast Guard estimated that the amount available for the Deepwater Project would range from $338 million (in year-of-expenditure dollars) in fiscal year 2002 to $547 million in fiscal year 2006.

If the Coast Guard proceeds with its current plans in issuing the RFP, contractors will be instructed to develop plans around a much higher funding stream than is available under the OMB budget targets. For example, the funding stream that the Coast Guard currently plans to use for the project ($350 million the first year and $525 million in subsequent years) is in 1998 dollars. Adjusted for inflation, this figure becomes $373 million in fiscal year 2002 (compared with OMB’s target of $338 million) and $569 million in fiscal year 2003 (compared with the target of $396 million). By the end of fiscal year 2006, the cumulative gap will total $496 million. While this shortfall may not seem so significant in the scheme of the overall budget, this amount is significant in the context of DOT’s total budget, especially given the competition among DOT agencies for available funding—a point that we discuss in more detail below. Figure 1 shows that the annual gap between planned funding and the amount available under OMB budget targets ranges from $35 million to $173 million.

Although the Coast Guard may be able to begin the project as planned at the level of funding provided for fiscal year 2002, the success of this first year may provide a false sense of security about how easy it will be to sustain projected funding levels. In fiscal year 2002, spending is relatively low compared with later years. Coast Guard officials said they plan to fully fund the contractor’s share of the planned amount and trim their own administrative expenses related to the project. In subsequent years, when planned payments to contractors rise much more steeply than amounts available, the gap may be far less manageable. In addition, the project’s first-year funding comes mainly from a source that will be soon exhausted. About $243 million of the amount proposed for the Deepwater Project in fiscal year 2002 would come from funds authorized in the 1998 Western Hemisphere Drug Elimination Act (P.L. 105-277), which will expire this year unless it is extended. OMB officials told us that it plans to request additional appropriations under this act in fiscal years 2003 to 2006.

Another concern is the potential effect of the Deepwater Project on other Coast Guard capital projects planned or under way. These other capital needs include, for example, modernizing communication equipment used to support search and rescue activities, and upgrading various shoreside facilities, such as boat stations and housing. The overall amount the Coast Guard now plans to spend on these projects is substantially less than the agency indicated in plans just a year ago. For example, in 2000, the Coast Guard’s planning documents proposed spending $475 million on nondeepwater projects in 2005. However, under the current plan, that spending level would drop by more than half, to about $196 million. In part, these proposals reflect the fact that some of the other capital projects, such as the buoy tender project, will be winding down, and the costs of other projects will be absorbed as part of the Deepwater Project. Rather dramatic reductions in other capital projects cannot be explained as easily. For example, estimated spending for improved shore facilities in fiscal year 2005 dropped from $147 million in last year’s plan to $59 million in the current plan. If estimates in the current plan hold true, fiscal year 2006 spending for nondeepwater projects will be at its lowest level in decades and call into question the validity of the agency’s estimates to maintain its current nondeepwater infrastructure. The presence of these other capital needs cannot be forgotten in assessing how ready the Coast Guard is to assume the risks of the Deepwater Project.

Future Funding for Deepwater Will Compete With Other Budget Priorities

The fiscal environment in which the Coast Guard must obtain funds for the Deepwater Project and other capital needs is further complicated by competition for funds with other DOT priorities. Obtaining additional funding for the Coast Guard within the DOT budget is likely to be difficult because of competition with other entities within the DOT appropriation, such as the FAA and Amtrak, for available discretionary funding. For example, recent action by the Congress limited FAA’s ability to use a separate funding source (the Airport and Airway Trust Fund) to fund FAA’s operations. As a result, funding for FAA’s operations now competes for the same limited DOT dollars on which the Deepwater Project would rely. FAA also expects its operating costs to increase to $7.4 billion by 2003, a 42-percent increase from 1998 levels. Similarly, Amtrak estimates that its capital needs alone will amount to about $1.5 billion annually through fiscal year 2020, part of which would come from the DOT budget.

Outside of DOT, the overall budget process is still driven by caps in discretionary spending. If these caps (which currently cover through fiscal year 2002) are extended as the administration has proposed, funding for the Deepwater Project would have to come from cuts in some other agency or program. The percentage increase in the Coast Guard’s budget request is among the largest of all federal agencies. However, the Coast Guard is basing its plans for the Deepwater Project on another major boost in funding beyond 2002. Thus, for all these reasons, sustaining the Deepwater Project at the funding level the Coast Guard is currently planning to use in its RFP appears to be a difficult task over a sustained period.

Developing Deepwater Around Lower Funding Scenario Would Help Reduce Risk

Our concern about this risk is not new. In several previous reports on the Coast Guard’s planning for the Deepwater Project, we expressed concern about the Coast Guard overestimating the amount of funding that would be available in the future for the project. The Coast Guard agrees that funding for the Deepwater Project is high risk and that it provides limited funding flexibility but believes it should keep its current approach of developing the project around the planned funding stream. Coast Guard managers believe that a deepwater system funded at planned levels provides the optimum system to meet deepwater requirements. The agency also believes that OMB budget targets could rise in the future and that the Congress could appropriate more funds to meet the agency’s capital needs. However, the potential risks are substantial and another strategy appears warranted. That strategy would be to develop a lower funding scenario around which the contractors can develop their proposals. If the project needs to be adjusted to a lower, more realistic funding stream, the time to do so is before the contract proposals are finished later this year.

Directing contractors to develop proposals around a lower funding scenario aligned with OMB targets would have several advantages. First, the Coast Guard would have greater opportunity to evaluate which proposal will produce the best value to the government within likely budget constraints. Second, the agency would be in a better position to hold the contractor accountable for delivering a system that meets original schedule and cost estimates if it selects the plan developed at the lower funding level. Using realistic funding expectations will reduce the risk of schedule stretch-outs and cost increases with the contractor in a sole- source environment after the contract is awarded—a situation in which the government’s leverage is reduced because it does not have the benefit of competition for obtaining a fair and reasonable price.

Any projection about likely funding levels for a project that lasts as long as the Deepwater Project will involve an element of uncertainty and risk. The Coast Guard’s current funding scenario exacerbates that risk. Because the Coast Guard has not yet issued its request for contractors to submit their best and final proposals, there is still time to mitigate the risk by identifying a lower funding stream that contractors should use in developing their proposals.

Ability to Control Costs in Project’s Later Years Remains Uncertain

The deepwater contracting approach that the Coast Guard adopted has never been tried on a contract this large extending over 20 or more years. At the time it was adopted, there was little evidence that the Coast Guard had analyzed whether the approach carried any inherent difficulties for ensuring best value to the government and, if so, what to do about them.We and others who are involved in reviewing this approach, such as OMB and the Office of Federal Procurement Policy, have expressed concerns about the potential lack of competition during the project’s later years and the reliance on a single contractor for procuring so much of the deepwater equipment. The Coast Guard is still conducting this analysis on its approach as it moves into phase 2 of the project and has delayed some of its key milestones to consider these concerns.

Little Evidence of Analysis Before Contract Approval Adopted

When the Coast Guard selected the contract approach in May 2000, it had not yet documented the risks involved or the degree to which this approach provided better value than other approaches. Contracting officials within the Coast Guard said their guidance from Coast Guard management had been to develop an approach that would (1) allow a single systems integrator to create a “system of systems” approach and (2) achieve potential improvements in operational effectiveness and minimize total ownership costs. Contracting officials told us that with these parameters in mind, they conducted a limited evaluation of several contracting alternatives by meeting informally with government and private sector officials about the Coast Guard’s proposed approach and meeting internally to discuss possible strengths and weaknesses of three approaches. Documentation detailing the basis for the decision—the depth of the analysis performed, the factors considered, the expertise sought (people contacted), and the compelling reasons why the current approach was chosen—was not recorded prior to its approval by Coast Guard acquisition officials. Without thorough documentation in this regard, the rigor of the Coast Guard’s analysis of the approach is unknown.

Approach Has Limitations for Ensuring That Costs Are Controlled

When we initially reviewed the Deepwater Project proposed contracting approach in March 2000, we expressed concerns about whether it could keep costs from rising and ensure good performance once the contract is awarded. We discussed the Coast Guard’s approach with contracting experts from both the public and private sector who, in addition to their concern about the Coast Guard’s ability to control costs, also raised concerns about certain management-related issues, which we cover later in this report. Presently, we focus on the cost-related issues of concern, namely the potential absence of competition for subcontracts in the project’s later years and the heavy reliance on a single-systems integrator to procure the entire system.

Potential Absence of Competition in the Project’s Later Years

OMB guidance recognizes the value of competition as a lever to keep contract costs down. The benefits of competition are present in the contract’s early years, as are other approaches for controlling costs. For the initial 5-year award term contract, prices for equipment and software to be procured are based on competition; and when the contract with the systems integrator is awarded, these prices will be fixed, according to Coast Guard officials. The Coast Guard also hopes to control costs by encouraging the use of commercially available (nondevelopmental) equipment. Prices for such equipment can be determined on the basis of previous orders from other buyers and by the use of fixed-price contracts.

Beyond the first 5-year award term contract, however, the benefits of competition are less certain. In a practical sense, the opportunity for competition in the project’s out years is diminished because the systems integrator will likely contract with those suppliers that were part of the team putting together the offer rather than opening the contract to a wider set of offerors. Coast Guard officials currently believe that a profit motive could drive the systems integrator to open competition to a wider set of offerors. Although this is possible, it would be easier to integrate equipment or subsystems acquired from a team member since equipment will be procured based on the plan developed by the team. A Coast Guard analysis of the same issue draws this same conclusion. We believe that this potential lack of competition reduces the normal marketplace control on price and subjects the Coast Guard to situations in which the supplier could potentially drive up project costs.

The Coast Guard is attempting to develop strategies for encouraging competition among suppliers, and thereby controlling costs, in subsequent 5-year award term contracts. One approach involves providing incentives for the systems integrator to submit “competitive proposals”—that is, proposals that are reasonably priced—-beyond the first few years of the contract. Contracting experts brought in by the Coast Guard discouraged this approach, saying such incentives usually have limited effectiveness. As a result, the Coast Guard now indicates it will evaluate the systems integrator’s performance in minimizing total ownership costs as part of its decision of whether to renew the systems integrator’s contract. By doing so, the Coast Guard hopes that this will encourage the systems integrator to have competition. At this point, it is not clear what effect this evaluation would have.

A second approach the Coast Guard plans to take is to negotiate a ceiling on the amount that will be paid for deepwater equipment in the 5-year period covered by a follow-on, award-term contract. This is a continuation of the approach being taken for the first 5-year contract. However, the ceiling could be waived if the project’s schedule or requirements are changed. Given the funding-related concerns discussed earlier, the potential for such changes cannot be easily dismissed. If such changes occur, the Coast Guard will rely on the systems integrator to negotiate prices with its vendors in a sole-source environment. Although doing so is a valid alternative for pricing a contract, a sole-source environment leaves little leverage in negotiations and therefore carries a higher risk of goods being overpriced. This approach also carries the burden of obtaining and reviewing cost and pricing data from suppliers and the systems integrator.

Reliance on Single Contractor

Another cost-related concern involves dependence on the systems integrator for a deepwater system that will take 20 or more years to acquire. This dependence is both one of the main strengths of the approach and one of its main weaknesses. On the positive side, if all aspects of the approach work well, the systems integrator will form a partnership with the Coast Guard and provide the technical expertise to assemble an integrated system and the continuity needed to bring a long- term project to a successful conclusion. However, the approach could establish the integrator as a monopoly supplier, substantially constraining the Coast Guard’s options or leverage. The Coast Guard could be in a weak position to negotiate aggressively on price because of its reluctance to take on the risks of increased costs and other problems associated with switching systems integrators. For example, if the systems integrator’s performance is unsatisfactory, a new systems integrator will have to step in to implement someone else’s partially completed design; or the Coast Guard will have to adopt a more traditional approach of buying individual classes of ships or aircraft, according to Coast Guard officials. The learning curve and other complications involved in such a midcourse adjustment could be dramatic and would probably be very costly.

Steps to Mitigate Contracting Risks Still Under Way

As our work progressed, we expressed our concerns to the Coast Guard immediately rather than waiting until the end of our review. As we raised these concerns, the Coast Guard took additional steps to study them. However, some of these efforts are still under way, and decisions have not been made on all specific measures to be incorporated into the acquisition plan and the RFP for the Deepwater Project.

In September 2000, we urged the Coast Guard to take several actions to deal with the risks of the contracting strategy it had selected. We suggested that the Coast Guard identify and evaluate all viable contracting approaches, discuss the approaches with contracting experts, and document the results. In particular, we stated that the Coast Guard should be open to options that would maximize the benefits of competition in later years while still maintaining the interoperability of the system. We also urged the Coast Guard to convene an independent panel of contracting experts from the government and private sector to review the proposed deepwater contracting approach or whatever approach the Coast Guard selected. We felt that given the contract’s uniqueness and the risks it poses, a rigorous review by a widely represented panel of experts was essential to both validate the Coast Guard’s approach, and recommend potential mitigating measures to strengthen it.

In December 2000, the Coast Guard proposed a limited peer review—one involving experts only from DOD and GAO and consisting of a 3-hour process (a 1-hour presentation on the contracting approach that the Coast Guard plans to use, followed by a 2-hour question-and-answer session). Because of our concerns about the limited nature of this approach, the Coast Guard—with our advice and assistance—expanded the panel of experts and adopted a more extensive, structured format. OMB officials share our concerns about the contracting approach, and they support the need for a peer review and a careful consideration of issues raised before the RFP and acquisition plan are finalized. We also urged the Coast Guard to provide the experts with key documentation, namely the acquisition plan and excerpts from the draft RFP prior to the meeting. Doing so would better ensure that the members of the panel have an objective basis for evaluating the Coast Guard’s contracting approach. However, the Coast Guard decided not to provide such documents to the panel members in advance, but instead provided them with selected excerpts from the acquisition plan. The entire acquisition plan and RFP was available for the panel upon request.

Subsequent to our discussions with Coast Guard management, the Coast Guard contracted with two outside consultants to review the proposed contracting approach for the Deepwater Project. One consultant was tasked to develop and recommend a contracting strategy for the deepwater system, given the Coast Guard’s requirement for an integrated “system of systems” solution. The consultant determined that the Coast Guard should continue with the approach it selected. However, citing cost increases and limited cost negotiation leverage as weaknesses, the consultant identified risk mitigation strategies, such as including in the RFP requirements for increasing competition over the mid to long term. A second consultant evaluated the draft RFP. He noted that this was one of the most complex contracts he had ever seen and suggested that it be simplified. For example, he suggested that the Coast Guard consider using incentives as part of the provisions of the award-term contract rather than as a separate item. He also observed that the success of the contracting approach is dependent on the Coast Guard receiving the planned funding stream.

To address the concerns raised by the consultants and to provide some time to respond to additional concerns that might be raised by the peer review, the Coast Guard has altered its planned date for issuing the deepwater RFP. The Coast Guard now plans to release the RFP in June 2001, or about 2 months later than its initial schedule. The Coast Guard is still responding to comments from its consultants and industry. Making necessary revisions to the RFP before giving it to contractors is important, because the RFP represents the contractual basis upon which the Coast Guard and the contractor will develop their relations. Also, changing the RFP after it has been issued could result in contractors having to amend their offers.

At this point, we do not know what changes the Coast Guard might adopt. Until adequate steps are in place to address concerns expressed by its consultants and by members of the peer review, we believe the risk related to cost control remains high. The Coast Guard’s success in this area also rests on how well it develops other sound strategies and options for managing potential problems. These strategies are discussed in the next section.

Overseeing the Acquisition Phase of the Deepwater Project Poses New Challenges

Another area of potential risk involves the overall management and day-to- day administration of the contract. In this regard, the Coast Guard’s performance during the planning phase has been generally excellent. During this phase, the Coast Guard took several innovative steps to establish and communicate what it wanted contractors to do, and it had adequate processes and trained staff in place to carry out the management tasks that needed to be done. As the project moves into the procurement phase, these challenges become more difficult, in large part, because the scope of work is so much greater and the contracting approach is unique and untried. It is too early to know if the Coast Guard can repeat the same strong performance on this much larger scale, because plans for managing and administering the deepwater contract are still being developed. The major challenges the Coast Guard faces involve developing and implementing plans for (1) establishing effective human capital practices, (2) having key management and oversight processes and procedures, (3) forming close relationships with subcontractors, (4) funding useful segments of the project, (5) tracking data to measure contractor performance, and (6) having an exit strategy and a contingency plan in the event of poor performance by the systems integrator.

Project Management During the Planning Phase Was Generally Excellent

In the planning phase of the project, the Coast Guard applied a number of “best practice” techniques recommended by OMB and others. Among them are the following: The Coast Guard gave contracting teams mission-based performance specifications, such as the ability to identify small objects in the ocean, rather than asset-based performance specifications, such as how large a cutter should be, and then it gave them leeway in deciding how to meet these specifications. Specifying performance criteria is the more traditional approach.

The Coast Guard established a management structure of Coast Guard and contractor teams for rapidly communicating technical information. Among other things, these teams assess each contractor’s evolving proposal to determine if it will meet contractual requirements and identify issues that could potentially have unacceptable effects. Communication mechanisms include an Internet Web site.

The Coast Guard highlighted the use of “open-system architecture” and emphasized the use of commercially supported products in the equipment to be acquired. This means that communication and computer equipment can be more easily replaced and upgraded without proprietary software or other unique requirements.

The Coast Guard also had effective procedures and a management structure in place for this phase of the project. Using a model developed by Carnegie Mellon University, we assessed the procedures and structure in eight key areas—planning, solicitation, requirements development and management, project management, contracting and oversight, evaluation, transition and support, and risk management. Within these 8 areas, we examined 112 key practices and found no significant weaknesses. In fact, the Coast Guard’s procedures and management structure for these eight areas were among the best of all the federal agencies we have evaluated using this model. This provides a good foundation for developing and implementing sound procedures for the next phase of the project; however, in many ways, the challenges will be more difficult.

Coast Guard Faces Difficult Challenges During the Procurement Phase

As the project moves from the planning phase to the procurement phase, the Coast Guard must ensure that it can perform project management and contract administration activities at a high level, given the complexity and scope of the contract and its uniqueness. Under the Coast Guard’s planned approach, the systems integrator will be responsible for program management required to implement the deepwater system, and the Coast Guard will continuously monitor the integrator’s performance. The Coast Guard plans to implement, or require the systems integrator to implement, many management processes and procedures based on best practices, but these practices are not yet in place. Because much work remains to be accomplished in this area, the full effectiveness of the Coast Guard’s approach cannot be assessed in the short term. The following are the key areas that will need to be addressed.

Effective Human Capital Practices

A critical element to the ultimate success of the project is having enough trained and knowledgeable Coast Guard staff to conduct management and oversight responsibilities. Project officials view this as a high-risk area and one of the most important aspects of the project. The Coast Guard hopes to have its full complement of staff needed for fiscal year 2002 by the time the contract is awarded. Currently, the Coast Guard has 69 personnel devoted to the Deepwater Project. According to project officials, the current project staff is highly qualified—most have advanced degrees in management, engineering-related, and other specialty fields. Moreover, the Coast Guard has made a conscious effort to maintain the continuity of its project staff by not rotating its military personnel on the project to new positions every 4 years as it normally does. In addition, the Coast Guard has assigned a Project Executive Officer to head the project.

Project officials have identified the need for 62 additional positions to manage the project beginning in fiscal year 2002. In addition, the officials plan to hire civilians with acquisition and contracting experience and to use support contractors for many activities. The Coast Guard is also in the process of developing a training plan for its project staff; it hopes to complete the plan later this year and ensure that all staff meet the training requirements for their respective positions by the time the contract is awarded.

Key Management and Oversight Processes and Procedures

Under its deepwater acquisition approach, the Coast Guard will rely heavily on the systems integrator to establish a management organization and systems necessary to manage the major subcontracts for deepwater equipment. The systems integrator will be required to apply an integrated product and process development approach, including teams consisting of Coast Guard, contractor, and major subcontractor personnel who are responsible for specific areas of the program. Also, the systems integrator will be responsible for developing key systems and processes, such as risk management, quality assurance, test and evaluation, and earned-value management systems.

In addition, the Coast Guard is developing a program management plan to oversee the systems integrator. The major components of this plan are project planning; organization; and detailed planning documents, including individual plans for contract management, information management, and financial management. Although the Coast Guard plans to complete the program management plan before the contract is awarded, project officials told us that some individual plans, such as configuration and integrated logistics support plans, are dependent upon the system selected and cannot be finalized until after the contract is awarded.

Close Relationships With Subcontractors

Because the use of major subcontractors to provide high-value equipment will be such an intricate part of the Deepwater Project, good relations and communications between the Coast Guard, the systems integrator, and the major subcontractors will be very important. Our past review of best practices on this issue suggests that leading organizations establish effective communications and feedback systems with their subcontractors to continually assess and improve both their own and supplier performance. These practices not only helped key subcontractors to fully understand the firms’ goals, priorities, and performance assessments, they also helped the firms to understand subcontractors’ ideas and concerns. Our experience in evaluating DOD acquisition programs showed that it was important to establish such relationships not only with prime contractors but with subcontractors as well. For example, supplier relationships on one program we reviewed reflected DOD’s traditional role of distancing itself from subcontractors. This role was traced, in part, to the fact that DOD had not articulated a particular subcontractor policy to guide program managers. We recommended—and DOD agreed—that DOD establish a policy and incorporate it into acquisition plans for major procurements.

The Coast Guard has developed no general policy on subcontractor relationships. Major subcontractors will be part of the integrated product and process development teams, and the Coast Guard plans to perform quality assurance activities at subcontractors’ facilities. However, according to project officials, the program management and quality assurance plans have not been completed, and it is not clear at this time what the quality and nature of the Coast Guard’s relationship with subcontractors will be.

Full Funding in Advance of Buying Equipment

OMB Circular A-11, Part 3, emphasizes that each useful segment (e.g. an entire ship or an entire aircraft) of a capital project should be fully funded in advance of incurring obligations. The Coast Guard has told its contractors to develop their deepwater schedules by using full funding of useful segments rather than incremental funding. Coast Guard contracting officials have said that they plan to obtain full funding for a ship or aircraft before proceeding with their procurement. However, if deepwater plans need to be adjusted due to a shortfall in funding or changes in program requirements, according to the officials, one option could be to develop requests that fund only part of a ship or aircraft. We found in a review of earlier Coast Guard budget justifications that the Coast Guard had proceeded with some capital projects before the amount of full funding was obtained. According to OMB, proceeding with such incremental funding could result in schedule delays or higher costs for capital projects. As the Coast Guard proceeds with the Deepwater Project, it should ensure that its budget requests are consistent with OMB guidelines on full funding of useful segments to avoid attendant delays and increased costs.

Accurate and Complete Data to Measure Contractor Performance

The Coast Guard plans to award follow-on, award-term contracts on the basis of factors such as improving operational effectiveness and minimizing total ownership costs. To measure the performance of the systems integrator in achieving these goals (as a basis for awarding the follow-on contracts), the Coast Guard will use a simulation model to measure improvements in operational effectiveness and will compare the contractor’s actual cost reductions with their proposed costs. According to Coast Guard officials, they will develop a new baseline for these factors on the basis of the winning contractor’s plans and the most current information on deepwater equipment after the contract is awarded in early 2002.

Coast Guard officials told us that they plan to use a subjective rating system to assess the contractor’s performance rather than use database benchmarks for improvements in operational effectiveness and total ownership costs. According to Coast Guard officials, setting such benchmarks may be difficult because performance data may reflect factors that did not result from actions of the contractor. For example, improved intelligence on drug smugglers could result in improvements in operational effectiveness. Also, changes in fuel costs could cause operational costs to increase. Because a host of factors could cause changes in these data, it will be important for the Coast Guard to carefully track these measures and accurately identify and segregate reasons for the changes that occur. Doing so would better show the results of significant federal investments in ships and aircraft.

Contingency Planning and Exit Strategies

Given the Coast Guard’s heavy reliance on a single systems integrator for so many facets of the Deepwater Project, the agency is at serious risk if— for whatever reason—the systems integrator does not perform as expected or decides to walk away from the project on its own. For example, if the systems integrator’s performance falls short of expectations, the Coast Guard will face numerous options, ranging from closer oversight to termination of the contract. Faced with these options, having a carefully thought-out contingency plan, which identifies and analyzes the implication of potential actions, would solidify the Coast Guard’s ability to respond effectively. Several high-level federal contracting officials echoed this position, saying that given the circumstances for this particular project, exit strategies and other means to deal with potential poor performance by the systems integrator were highly desirable.

In the extreme case—where the contractual relation with the systems integrator is terminated—an exit strategy identifying possible alternatives, consequences, and transition issues would be important. In this regard, contracting officials with the project told us that the contract will provide several “off-ramps” and that the Coast Guard has basically two options if it were to terminate the systems integrator: (1) obtain a new systems integrator and a new set of subcontractors as well and (2) revert to the traditional “stovepipe” procurement approach of procuring a single class of vessels and aircraft at a time. These officials said that from a project management standpoint, having a strategy to deal with options like these is important; and the agency is currently documenting, with assistance from a contractor, the pros and cons of each exit strategy. However, the officials noted that specific, detailed plans to implement the options would not be developed until it was known that the Coast Guard planned to terminate the contract.

Use of Off-the-Shelf Technology Minimizes Risks, but Effective Means to Assess Unproven Technology Is Lacking

The risks associated with incorporating new unproven technology into the first part of the Deepwater Project are minimal, in part, because of the Coast Guard’s emphasis that industry teams use technology that has already been proven in similar applications. Our main concern is the absence of criteria to measure the risk of the new technology that needs to be developed, both now and in the project’s later years.

Coast Guard’s Approach Conforms With Best Practices for Technology Development

Too little assessment of the risks associated with developing new technology has caused problems on many acquisition projects, both in government and the private sector. OMB’s Capital Programming Guide (A- 11) states, “Probably the greatest risk factor to successful contract performance is the amount of technology development that is planned for the procurement.” Minimizing a technology’s unknowns and demonstrating that it can function as expected significantly reduce such risk. We have found that leading commercial companies use disciplined processes to demonstrate—before fully committing to product engineering and development—that technological capability matches project requirements. Waiting to resolve these problems can greatly increase project costs—at least 10-fold if the problems are not resolved until product development, and as much as 100-fold if they are not resolved until after production begins.

The Coast Guard has taken steps to minimize these risks. One major step was to emphasize in contracting documents to industry teams that, to the maximum extent possible, proposed assets, systems, equipment, and components are to be nondevelopmental or commercially available (off- the-shelf) items. Our review showed that the teams’ preliminary proposals included many commercial off-the-shelf and nondevelopmental items currently operating in the commercial or military environment. However, some proposed equipment included developing technology that has not yet been proven. Generally, these developing technologies are at the prototype level and are undergoing performance testing and evaluation prior to contract award to commercial and military customers.

The Coast Guard’s steps are helping to keep the risk of unproven near- term technology at a low level. We measured the maturity level for the project’s most critical near-term technologies (those introduced in the first 7 years of the project), using an approach developed by the National Aeronautical and Space Administration (NASA). We applied this process, referred to as technology readiness levels (TRL), to 18 technologies identified as critical by the 3 contractor teams and the Coast Guard. We determined—and the Coast Guard concurred—that by the time the contract is awarded, 16 of the 18 are expected to be at a level of acceptable risk. The remaining two technologies will be slightly higher in risk; but in one case, an early prototype is being tested, and in the other, a proven backup system has been identified that, if needed, could replace the technology with no effect to the project’s cost, schedule, or performance. Entering phase 2 of the project with critical technologies at a high level of maturity or with proven backup systems significantly lowers risk and the likelihood of delays, which in turn helps to control program costs.

Criteria to Adequately Monitor and Assess Technology Maturity Are Lacking

Although technological risks appear minimal in the near term, the Coast Guard lacks criteria for assessing the maturity of technology in the longer term. The Coast Guard has a risk-management plan in place, as well as a process to identify, continuously monitor, and assess technology risks; and the resources the Coast Guard expects to commit to the task during phase 2 appear to be adequate. What the process lacks, however, is uniform and systematic criteria for judging the level of technology maturity and risk, such as the TRL ratings in the approach we adopted from NASA. In contrast, since January 2001, DOD has required the use of TRL criteria as a tool for measuring the technology readiness of its procurement projects.

Such criteria are important for monitoring both continued development of the technologies we examined and the development of other technologies that will not be used until later in the project. As of July 2000, when we completed our TRL assessment, half of the 18 deepwater key technologies we reviewed were still below the maturity level considered an acceptable risk for entering production. Before the contract is awarded, the Coast Guard must assess the readiness of these technologies. In addition, the industry team proposals include numerous technologies that are planned for deepwater system introduction from 2009 to 2020—well after contract award. Many of these future technologies will not be proven at contract award and will need to be assessed for technology risk before acceptance. The Coast Guard plans to have a test and evaluation master plan in place by June 2001, but it is not planning to include a requirement for using TRL criteria to measure technology readiness in that plan.

Conclusions

The Coast Guard’s acquisition strategy and contracting approach for the Deepwater Project are innovative. The agency plans to use the full flexibility provided by congressional reforms of the federal acquisition process to avoid the all too frequent failures of major federal acquisitions in the past. Despite the numerous commendable innovations during the concept development phase, we remain concerned that considerable risks remain with its chosen approach for the procurement phase of the acquisition.

The Coast Guard’s contracting approach for the production phase of the deepwater acquisition is unique—relying on a single contractor to manage, build and integrate the modernization of its entire deepwater fleet over a period likely to exceed over 2 decades. The key promise of the approach is achievement of a fully integrated system that both maximizes improvements in operational effectiveness and minimizes total ownership costs (including not only the acquisition, but operation, maintenance, and support costs of the entire system over a 40-year period).

While we recognize the merit of exploring innovative and even unique approaches, we believe the selected approach puts at risk precisely the purported benefits of the approach—that is, maximizing operational effectiveness and minimizing operational costs. Development of this unique and untried approach on such a large scale and for an acquisition so critical to the Coast Guard’s ability to perform every aspect of its deepwater mission puts a heavy burden on the Coast Guard. Not only would it be reasonable to expect a rigorous effort to identify and mitigate all the major potential risks associated with a totally new approach, but the Coast Guard would also need to ensure that other approaches were fully evaluated. Unfortunately, we found that the Coast Guard has yet to accomplish either. At our urging, the Coast Guard has only recently sought to set up a systematic effort to identify and mitigate risks associated with its chosen approach and the evaluation of alternative approaches remains limited and poorly documented.

We remain concerned that the Coast Guard will soon be making critical decisions regarding the Deepwater Project, namely issuing the RFP in less than 2 months and awarding a contract to procure deepwater equipment in less than a year. Yet, significant risks still exist, and the Coast Guard has not completed actions to fully address them. The unique contracting approach of relying on a single systems integrator to manage, acquire, and integrate all Deepwater assets and capabilities poses two major risks, both of which still remain. First, the agency’s choice of a contracting approach is now inextricably tied to a projected deepwater funding level of over $500 million annually for the next 2 to 3 decades. Attaining sustained funding for the project at this level looms as the major potential problem. By choosing to proceed with a funding scenario that appears to be unrealistically high in the face of budget projections that are substantially less, the Coast Guard is increasing the risk that the project will incur future cost increases and schedule stretch-outs. Second, the Coast Guard’s reliance on a single contracting team raises serious questions regarding the Coast Guard’s ability to control costs and ensure performance once the contract is awarded. Their strategy for adequately controlling costs in the project’s later years is still being worked out and requires careful attention before the RFP is issued. Similarly, the Coast Guard is still developing plans for managing the contract, and much remains to be done. These are risks that need to be well understood and resolved before the RFP is issued. Moving ahead before addressing the major risks and evaluating options for addressing them, potentially including an evaluation of alternative approaches, would be unwise.

The Coast Guard’s acquisition approach for the Deepwater Project—and its reliance on a large and sustained funding level over a long period— makes the Congress’ next decision on the project crucial as well. This decision goes far beyond deciding what Coast Guard equipment needs to be replaced or modernized. The Congress is in effect being asked to provide the first installment based on the Coast Guard’s spending plan for the project, which is essentially dependent on a continuous funding stream in excess of $500 million annually for decades. Allowing the project to proceed as planned and then later reducing that funding level significantly would result in higher system costs and reduced system performance. We think this is the central risk posed by the current approach, and that the Congress needs to make its decision about providing funding for the project this year with clear knowledge that the Coast Guard’s chosen contracting strategy depends heavily on a sustained high level of funding for at least the next 20 years.

Recommendations for Executive Action

We recommend that before the Coast Guard issues the RFP for the Deepwater Project, the Secretary of Transportation should ensure that a realistic level of funding, based on OMB budget targets, the Coast Guard’s capital planning process, and congressional guidance is incorporated into the RFP and used by contractors as the basis for designing their proposed systems; and direct the Commandant of the Coast Guard to carefully consider and incorporate recommendations, if any, made by the peer review panel into the deepwater acquisition plan and RFP or if the peer review panel finds serious and unmitigated risks in the Coast Guard’s approach, evaluate alternative contracting strategies that could address the risks.

Before the Coast Guard signs a contract with the systems integrator for the Deepwater Project, we recommend that the Secretary of Transportation should direct the Commandant of the Coast Guard to address the following issues: complete development of the Program Management Plan, including plans and procedures to (1) facilitate relations with subcontractors, (2) ensure that the project is adequately staffed and that the staff is properly trained to perform their respective project management responsibilities, and (3) cover actions to be taken in the event that the Coast Guard decides not to continue its contract with the systems integrator; complete plans for ensuring that annual budget requests for the Deepwater Project are for useful segments and that a mechanism is in place for reporting to OMB and the Congress, as part of its annual budget submission, the progress that is made in achieving baseline goals of minimizing costs and improving operations due to investments in funding the Deepwater Project; and select a process, such as the technology readiness levels approach, for assessing the technology readiness of equipment and major systems to be delivered.

Matters for Congressional Consideration

The success of the contracting approach the Coast Guard selected for the Deepwater Project relies heavily on the Coast Guard being able to sustain the funding level around which the contractor’s proposal is based. Substantial and prolonged funding below that level will lead not only to cost increases and schedule slippages, but also to situations in which the Coast Guard’s ability to achieve its missions may be jeopardized. To avoid these situations, the Congress should have the opportunity to weigh in on the affordability of the project before the contract is awarded. Therefore, the Congress may wish to direct the Secretary of Transportation to (1) ensure that any funding scenario included in the RFP is based on OMB budget targets as well as discussions with appropriate congressional committees, (2) submit a report to Congress justifying the funding scenario and explaining any variations from the funding projections of OMB, and (3) wait 30 calendar days from submission of the report before issuing the RFP.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Transportation and the Office of Management and Budget for their review and comment.

In commenting on our draft report, DOT disagreed with our recommendation to incorporate more realistic levels of funding for the project into the RFP based on OMB’s budget targets. In support of its position, DOT noted that OMB out-year funding targets have been converging with estimated project requirements during the last year, and it believes that OMB targets will change in the future to better match project requirements of $500 million annually. DOT’s position in this regard is counter to good capital planning and OMB guidance that says that agencies should plan projects within available funding levels. As noted in the report, the Coast Guard faces the real possibility of a cumulative funding shortfall of almost half a billion dollars, or over 20 percent of the total funding needs for the project’s first 5 years. Ultimately, by the Coast Guard’s own admission, funding levels significantly below project requirements would most likely lead to cost increases and schedule slippages and jeopardize the agency’s ability to achieve its missions.

DOT agreed with two recommendations and did not comment on two others. The agency agreed to evaluate and incorporate into the RFP as appropriate recommendations from the peer review panel on its contracting approach for the Deepwater Project. Also, the agency agreed to complete development of the Program Management Plan prior to awarding the contract for phase 2. DOT had no comment on two other recommendations, which focused on (1) ensuring that its annual budget requests are for useful segments and that a mechanism is in place for reporting to OMB and the Congress on the progress in achieving baseline project goals and (2) selecting a process for assessing the technology readiness of equipment and major systems to be delivered. DOT’s written comments and our response are in appendix V.

We met with officials from OMB, including the Chief, Transportation Branch. OMB concurred with our recommendations but believed that additional actions may be warranted. OMB has concerns about the deepwater acquisition strategy and believes that a broader evaluation of alternative strategies is needed. The agency indicated that the Coast Guard has chosen an approach that relies on a required funding level each year, and OMB has the same concerns that we do about the potential impact on the project if funding does not materialize as expected. OMB is also concerned that this approach sets up a situation where the administration and the Congress would have to fund the project in later years at the planned level, regardless of other competing priorities. Essentially, OMB believes that the deepwater funding strategy transfers the risk of program failure to external sources, such as the Congress. According to OMB, future funding levels cannot be guaranteed, and it would be inappropriate for the Coast Guard to use funding levels in the RFP that are not consistent with OMB’s targets. Under the current acquisition approach, if sustained funding is substantially less than planned, the Coast Guard would have to rebaseline the project in a sole-source environment, a situation that could increase project costs even further. Finally, OMB raised these concerns at the peer review panel meeting; however, OMB is not optimistic that the Coast Guard will sufficiently recognize and adequately address its concerns prior to issuing the RFP. We generally share OMB’s concerns and have made many of the same points throughout our report.

We plan to provide copies of this report to the Honorable Norman Y. Mineta, Secretary of Transportation; Admiral James M. Loy, Commandant of the Coast Guard; and the Honorable Mitchell E. Daniels, Jr., Director, Office of Management and Budget. We will also send copies to others upon request.

If you or your staff have any questions about this report, please call me at (202) 512-2834 or Randall Williamson at (206) 287-4860. Other key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This report examined the major risks associated with the Deepwater Project and the progress the Coast Guard has made in addressing them. Our work focused on four risks: (1) planning the project around annual funding levels far above what the administration has told the Coast Guard it can expect to receive, (2) keeping costs under control in the contract’s later years, (3) ensuring that procedures and personnel are in place for managing and overseeing the contractor once the contract is awarded, and (4) minimizing potential with developing unproven technology.

To assess the funding risk, we reviewed OMB Circular No. A-11, Part 3 (Planning, Budgeting, and Acquisition of Capital Assets); OMB’s Capital Programming Guide, the Coast Guard’s 5-year capital plan; and the agency’s past budget requests for capital projects. We also reviewed various deepwater planning documents, including the risk management plan, the draft acquisition plan, and requests for proposal. We reviewed prior Coast Guard appropriations and pertinent laws affecting the Coast Guard’s budget. We also reviewed DOT Inspector General reports on the Deepwater Project. We interviewed DOT and Coast Guard officials involved in forming the Coast Guard’s budget, including the Coast Guard’s Director of Resources. We also interviewed OMB budget officials and officials from the Office of Federal Procurement Policy.

To assess the risks of controlling costs, we reviewed the Federal Acquisition Regulation; OMB Circular A-11, Part 3, (Planning, Budgeting, and Acquisition of Capital Assets); and OMB’s Best Practices for Multiple Award Task and Delivery Order Contracting. We reviewed the Coast Guard’s draft acquisition plan and RFP for the deepwater phase 2 contract, reports that the Coast Guard received from consultants it hired to evaluate its acquisition strategy, and numerous Coast Guard documents regarding how the agency planned to acquire deepwater assets. We interviewed numerous contracting officials, including, Coast Guard contracting officials, officials from OMB’s Office of Federal Procurement Policy, the former Deputy Under Secretary of Defense for Acquisition Reform, the Deputy Director for Defense Procurement, the Chief of the Internal Revenue Service’s Contracting Branch, and an official of a private consulting firm. We also drew from our extensive agencywide contracting experience in reviewing DOD and other agency procurements.

To determine the risk involved in managing the contract, we assessed the Coast Guard’s project management during the planning phase of the Deepwater Project and identified challenges the Coast Guard will face during the procurement phase. To identify best practices in contract management and administration, we reviewed OMB Circular No. A-11, Part 3, and drew from our extensive agencywide contacting experience in reviewing DOD and other agency procurements. We reviewed the Coast Guard’s Project Management Plan, Risk Management Plan, and other management plans to identify the Deepwater Project’s organizational structure and key management procedures used during the planning phase. We assessed the effectiveness of these procedures and structure using Carnegie Mellon University’s Software Engineering Institute’s Software Acquisition Capability Maturity Model® and its Software Capability Evaluation method. Although the model is specifically designed to determine software acquisition process maturity, its application can be used for the acquisition of any type of asset (ships, aircraft, etc.). The model ranks organizational maturity according to five levels. Maturity levels 2 through 5 require the verifiable existence and use of certain acquisition processes, known as key process areas. Satisfying the requirements of maturity level 2 demonstrates that an organization has the policies needed to manage a project and the procedures needed to implement those policies. We evaluated the acquisition processes for two Deepwater Project matrix product teams against all seven level-2 areas (planning, solicitation, requirements development and management, project management, contracting and oversight, evaluation, and transition and support) and one level-3 area (risk management). Within these 8 key process areas, we examined 112 key practices to determine their strengths or weaknesses. We reviewed the Coast Guard’s draft acquisition plan and RFP for the Deepwater Project phase 2, comments that the Coast Guard received from the consultants it hired to evaluate its acquisition strategy, and other documents to identify how the agency plans to manage and administer the procurement phase of the Deepwater Project. We discussed these management plans with Coast Guard contract and Deepwater Project officials, the DOD Deputy Director for Defense Procurement, the Chief of the Internal Revenue Service’s Contracting Branch, and a representative from a private consulting firm.

To assess the risk of using new technologies, we asked each of the three competing deepwater contracting teams to first develop a list of the most critical technology and keystone systems being proposed as “near-term” deepwater contract deliverables to be introduced during the first 7 years (2002 through 2008) after contract award. Eighteen technologies and systems were identified, including assets and components representing deepwater aviation, surface, and command, control, communications, computers, intelligence, surveillance, and reconnaissance concept solutions. We then asked the contracting teams to assess “technology readiness” for each of the items they identified on their lists using NASA’s technology readiness level (TRL) criteria. TRLs provide a standard definition of nine levels of technology maturity that can be used to measure technology readiness, regarding the type of demonstration that must be achieved; the scale (form, fit, and function) of the asset; and the operational environment in which demonstration is performed. We asked the contracting teams to score technology readiness at three points in the deepwater acquisition process—July 2000, April 2001, and January 2002. We focused our analysis on the technology readiness level at the date of contract award, January 2002. We independently met with program managers from each of the three industry teams to discuss the status of each technology/keystone system, identify the rationale for the initial TRL score assessment, and determine whether adjustments in the TRL score were necessary. On the basis of these discussions, we made adjustments to the initial TRL scores that the competing contractors agreed were consistent with the TRL criteria. We then crosswalked the TRL scores to project risk criteria established by the Air Force Research Laboratory that predicts project risk on the basis of technology readiness at program decision points. Specifically, the Laboratory established that a technology/key system should be at TRL 7 at the time the decision is made for a program to enter the Engineering and Manufacturing Development Phase—a phase we believe is comparable to the January 2002 deepwater contract award for “near-term” technology and keystone systems.

Appendix II: Current Deepwater Cutters and Aircraft

Appendix II: Current Deepwater Cutters and Aircraft This is the largest multipurpose cutter in the fleet. It has a planned crew size of 167, a speed of 29 knots, and a cruising range of 14,000 nautical miles. The Coast Guard operates each cutter for about 185 days a year, and it can support helicopter operations. This cutter has a planned crew size of 100, a speed of 19.5 knots, and a cruising range of 10,250 nautical miles. The Coast Guard operates each cutter for about 185 days a year, and it can support helicopter operations. This cutter has a planned crew size of 75, a speed of 18 knots, and a cruising range of 6,100 nautical miles. The Coast Guard operates each cutter for about 185 days a year, and it can support operations of short-range recovery helicopters. This patrol boat has a crew size of 16, a speed of 29 knots, and a cruising range of 3,928 nautical miles. The Coast Guard operates each for about 1,800 hours a year. The 213-foot medium-endurance cutter, commissioned in 1944, has a planned crew size of 64. The 230-foot medium-endurance cutter, commissioned in 1942, has a planned crew size of 106. The 282-foot medium-endurance cutter, commissioned in 1971, has a planned crew size of 99.

This is the largest aircraft in the Coast Guard’s fleet. It has a planned crew size of seven, a speed of 290 knots, and an operating range of about 2,600 nautical miles. The Coast Guard operates each of these aircraft for about 800 hours every year. This is the fastest aircraft in the Coast Guard’s fleet. It has a planned crew size of five, a speed of 410 knots, and an operating range of 2,045 nautical miles. The Coast Guard operates each for about 800 hours a year. This helicopter is capable of flying 300 nautical miles off shore, remaining on scene for 45 minutes, hoisting six people on board, and returning to its point of origin. The Coast Guard operates each for about 700 hours a year. It has a planned crew size of four, a maximum speed of 160 knots, and a maximum range of 700 nautical miles. This helicopter is capable of flying 150 nautical miles off shore. It has a crew allowance of three, a maximum speed of 165 knots, a maximum range of 400 nautical miles, and a maximum endurance of 3.5 hours. The Coast Guard operates each for about 645 hours a year.

Appendix III: Estimated Service Life of the Coast Guard’s Deepwater Cutters and Aircraft

In our 1998 study on the Deepwater Project, we found that the Coast Guard had substantially understated the service life of its aircraft and, to a lesser extent, its ships. For example, in its project justification prepared in 1995, the Coast Guard estimated that its aircraft would need to be phased out starting in 1998. However, in 1998, a Coast Guard study showed that with proper maintenance and upgrades, its aircraft would be capable of operating until at least 2012 and beyond. Also, a September 1999 study revised earlier estimates and concluded that the Coast Guard’s deepwater cutters have a service life until 2007 and beyond, assuming that adequate funds remain available for maintenance support and service life upgrades. Shown below are the differences in the service life of its deepwater ships and aircraft between the initial estimates (in the1995 justification) and later studies.

Appendix IV: List of the Members of Each Contracting Team

During Phase 1 of the Deepwater ProjectLitton/Avondale Industries (Systems Integrator) Boeing-McDonnell Douglas Corporation John J. McMullen & Associates, Inc. DAI, Inc. Raytheon Systems Company Lockheed Martin Naval Electronics and Surveillance Systems (Systems Integrator) Lockheed Martin Aeronautical Systems Lockheed Martin Electronics Platform Integration - Oswego, NY Lockheed Martin Global Telecommunications Lockheed Martin Management and Data Systems Sanders, A Lockheed Martin Company Litton/Ingalls Shipbuilding Litton/PRC M. Rosenblatt & Son Bell Helicopter Textron, Inc. Halter-Bollinger Joint Venture, L.L.C. Acquisition Logistics Engineering L-3 Communications East PROSOFT Whitney, Bradley and Brown, Inc.

Science Applications International Corporation (Systems Integrator) Marinette Marine Corporation Sikorsky Aircraft Corporation Soza & Company, Ltd. Bath Iron Works AMSEC Fuentez Systems Concepts, Inc. Gibbs & Cox, Inc.

Appendix V: Comments From the Department of Transportation

GAO Comments

The following are GAO’s comments on DOT’s letter dated April 19, 2001. 1. Our report notes that the Coast Guard took many innovative steps and recognizes that the agency’s procedures and management structure for the planning phase of the Deepwater project were excellent. While its management during the planning phase provides a solid foundation for the project, the acquisition phase presents considerably tougher challenges. By almost everyone’s assessment, the acquisition strategy is a high risk, untried approach to procure deepwater assets. Whether the Coast Guard has adequately addressed these risks will not be known for years to come. Furthermore, the Coast Guard’s handling of remarks and suggestions made by members of the peer review are largely unknown at this point. 2. The contracting approach lacks flexibility in several key areas. First, it requires sustained funding at planned levels of more than $500 million for 2 or more decades. Second, it offers no true means to ensure competition for major components as a lever to minimize costs. Third, if planned funding levels are not realized, it opens the door to added costs because the Coast Guard would have to renegotiate costs and delivery dates—all in a sole source environment. Finally, changing the systems integrator after the contract is awarded—while doable— would likely be costly both in terms of dollars and delays in the project. 3. The added dollars expected from the Western Hemisphere Drug Elimination Act has allowed the budget targets to increase substantially from prior year (fiscal year 2001) targets. However, OMB told us that had it not been for the act, the large increase in the Coast Guard’s targets for capital projects would have been difficult to achieve given the budgetary environment. While targets may increase somewhat in future years, any large increases would require new funding sources or shifts in funding from other entities, such as FAA and Amtrak, which also have critical capital needs. Already, the funding requirements for the project are almost half a billion dollars more than OMB budget targets through 2006. Given the uncertainty of future funding, it would be unwise and fiscally imprudent for the Coast Guard to blindly proceed with an RFP that contains a planned funding stream of $500 million, hoping that funding at planned levels will materialize later. OMB echoed our position on this issue. 4. While the Coast Guard has the flexibility to alter project plans based on reduced funding levels in future years, the Coast Guard would likely pay dearly for this. The Coast Guard recognizes this but steadfastly opposes including a lower, more realistic funding level in its RFP. It has essentially rejected our concerns and those of OMB in this area and has adopted a position that runs counter to sound “best practices” for capital planning that are based on widely-accepted OMB guidance. 5. The Coast Guard’s characterization of the peer review panel’s deliberations and findings are overly optimistic and overstate the positive results from the panel. Our review of the transcript of the panel’s deliberations showed that there was not the unanimous consensus among panel members on the efficacy of the acquisition approach that was portrayed by the Coast Guard. For example, the panel member from the Office of Federal Procurement Policy voiced numerous concerns about whether a thorough and honest risk analysis of the acquisition approach had been done and whether adequate mitigation and management plans are in place. Another member also echoed this position, while another remarked that much work is needed before the RFP should be issued to contractors. We believe that such concerns by panel members do indeed refer to potentially serious and unmitigated risks that should not be dismissed lightly. In addition, given that panel members were not given the RFP or the acquisition plan prior to the panel meeting, we question the thoroughness of the panel’s results and the depth to which it explored key questions. It is evident from the transcript of the panel discussions and our observations of the proceedings that panel members may not have had a good grasp or understanding of many issues in the depth necessary to make informed observations and suggestions. For example, one panel member remarked in his summary at the end of the panel discussion that there was an information void on some issues when the panel began discussions, and having more detailed information on the acquisition strategy ahead of time would have been useful. OMB officials who observed the peer review session told us that they felt the same way. Moreover, the panel members were not asked to determine whether this approach represents the “best approach among all possible alternatives,” nor were panel members given the time or the information necessary to make such a determination. 6. To provide funding for the Deepwater Project, the Coast Guard will likely have to keep funding for other capital projects at levels which would be substantially lower than levels experienced over the last decade or more. It is unrealistic to believe that other non-deepwater capital needs will be minimal for the entire duration of the Deepwater Project. The DOT Office of Inspector General, for example, has recently identified millions of dollars of potential capital projects associated with the Coast Guard’s search and rescue program. Also, in its current fiscal year 2002 capital plan, the Coast Guard may have significantly understated amounts needed for information technology and other projects. For example, the current plan projects information technology funding needs of only $3 million in 2005; its capital plan of just a year ago cited information technology project needs of $31.4 million in 2005. Similarly, estimates of funding needs for shore facilities were $128.8 million in the 2001 plan and only $58.7 million in the fiscal year 2002 plan. Either the Coast Guard grossly overstated its non-deepwater needs in the fiscal year 2001 plan or it cut deeply into these projects for the fiscal year 2002 plan to accommodate funding for the Deepwater Project. Regardless, this leaves serious questions about whether the Coast Guard is understating funding needs for non- deepwater projects to give the appearance that the Deepwater Project funding needs can be met in the next 5 years. 7. While the Coast Guard has provisions in the RFP that allow it to exit the contract if price or performance is unsatisfactory, the practical reality is that changing the systems integrator will be costly, and there is a natural reluctance for an agency to do so. Members of the peer review panel remarked similarly on this issue. In addition, complete, reliable data on total ownership costs and operational effectiveness may be absent, especially in the project’s early years, making those measures less effective as a means to evaluate contractor performance. Members of the peer review panel made this point as well. Also, the inclusion of contract incentives does not guarantee competition will exist among subcontractors. The panel did not reach unanimous consensus that such incentives would necessarily be effective in this regard, as the Coast Guard contends. 8. The Coast Guard did not comment on two recommendations that need to be addressed. Developing an effective assessment tool to evaluate the technology maturity of major equipment and components is critical to keep a tight rein on costs. Also, ensuring that future budget requests for deepwater components are for useful segments is essential. OMB strongly concurred with our view on these issues. Finally, keeping the Congress appraised of progress being made in achieving the baseline goals of minimizing costs and improving operations is vital as a basis for holding the Coast Guard accountable to the Congress and the administration for the significant investment in the project.

Appendix VI: GAO Contacts and Staff Acknowledgements

GAO Contacts

Acknowledgements

In addition to those named above, Marie Ahearn, Neil Asaba, Naba Barkakati, Alan Belkin, Christine Bonham, Sue Burns, John Christian, Tom Collis, Ralph Dawn, Paul Francis, David Hooper, Richard Hung, Matt Lea, Sterling Leibenguth, Lynn Musser, Madhav Panwar, Colleen Phillips, David Robinson, Katherine Schinasi, Stanley Stenersen, Mike Sullivan, and William Woods made key contributions to this report.