Contract Management: Coast Guard's Deepwater Program Needs Increased Attention to Management and Contractor Oversight
Published: 2004-03-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Coast Guard's Deepwater program, the largest acquisition program in its history, involves modernizing or replacing ships, aircraft, and communications equipment. The Coast Guard awarded the Deepwater contract to Integrated Coast Guard Systems (ICGS) in June 2002. The Coast Guard estimates the program will cost $17 billion over a 30-year period. ICGS is a system integrator, with responsibility for identifying and delivering an integrated system of assets to meet the Coast Guard's missions. GAO was asked to assess whether the Coast Guard is effectively managing the Deepwater program and overseeing the contractor and to assess the implications of using the Deepwater contracting model on opportunities for competition.
section_title: What GAO Found
paragraphs: Over a year and a half into the Deepwater contract, the key components needed to manage the program and oversee the system integrator's performance have not been effectively implemented. Integrated product teams, the Coast Guard's primary tool for overseeing the system integrator, have struggled to effectively collaborate and accomplish their missions. They have been hampered by changing membership, understaffing, insufficient training, and inadequate communication among members. In addition, the Coast Guard has not adequately addressed the frequent turnover of personnel in the program and the transition from existing to Deepwater assets. The Coast Guard's assessment of the system integrator's performance in the first year of the contract lacked rigor. For example, comments from the technical specialist responsible for monitoring the design and delivery of ships were not included in the evaluation scores. Further, the factors that formed the basis for the award fee determination were unsupported by quantifiable metrics. Despite documented problems in schedule, performance, cost control, and contract administration, ICGS received a rating of 87 percent, resulting in an award fee of $4.0 million of the maximum $4.6 million annual award fee. Further, the Coast Guard has not yet begun to measure the system integrator's performance on the three overarching goals of the Deepwater program--operational effectiveness, total ownership cost, and customer satisfaction. Its original plan of measuring progress on an annual basis has slipped, and Coast Guard officials have not projected a time frame for when they will be able to hold the contractor accountable for progress against these goals. This information will be essential to the Coast Guard's decision about whether to extend ICGS's contract after the first 5 years. Competition is critical to controlling costs in the Deepwater program and a guiding principle of Department of Homeland Security acquisitions. Concerns about the Coast Guard's ability to rely on competition as a means to control future costs contributed to GAO's description of the Deepwater program in 2001 as "risky." Three years later, the Coast Guard has neither measured the extent of competition among suppliers of Deepwater assets nor held the system integrator accountable for taking steps to achieve competition. Deepwater's acquisition structure is such that the two first-tier subcontractors have sole responsibility for determining whether to hold competitions for assets or to provide these assets themselves. The Coast Guard has taken a hands-off approach to "make or buy" decisions made at the subcontractor level. As a result, questions remain about whether the government will be able to control costs.

section_title: Letter
paragraphs: The Coast Guard is undergoing a major transformation through its  Deepwater program, the largest acquisition project in its history. The  program involves modernizing or replacing cutters, aircraft, and  communications equipment used for missions that generally occur beyond  50 miles from shore. These missions include drug interdiction, alien  migrant interdiction, search and rescue, overseas port security and  defense, and offshore inspection of foreign vessels bound for U.S. ports.  The three goals of the Deepwater program are to maximize operational  effectiveness, minimize total ownership cost (TOC), and satisfy the  customer. On March 1, 2003, the Coast Guard became part of the  Department of Homeland Security (DHS), and the Deepwater program,  estimated to cost $17 billion over a 30-year period, is one of the largest  acquisitions in the department.
In June 2002, the Coast Guard awarded a contract to Integrated Coast  Guard Systems, LLC (ICGS) as the system integrator for Deepwater to  develop and deliver an improved, integrated system of ships, aircraft,  unmanned aerial vehicles, command, control, communications, computer,  intelligence, surveillance and reconnaissance (C4ISR), and supporting  logistics. The contract has a 5-year base period with five additional 5-year  options. ICGS is a business entity jointly owned by Northrop Grumman  Ship Systems (Northrop Grumman) and Lockheed Martin Corporation  (Lockheed Martin). Northrop Grumman and Lockheed Martin are the two  first-tier subcontractors for the Deepwater program. They, in turn, provide  Deepwater assets or award second-tier subcontracts for the assets. Figure  1 depicts the relationship of the various parties involved in executing the  Deepwater program.
The Deepwater contract is performance-based, meaning that the Coast  Guard has specified the outcomes it is seeking to achieve and has given  the system integrator responsibility for identifying and delivering the  assets needed to achieve these outcomes. In 2001, we described the  Deepwater project as “risky,” due to the unique, untried acquisition  strategy for a project of this magnitude within the Coast Guard. We  highlighted concerns about the Coast Guard’s ability to ensure that  procedures and personnel would be in place to manage and oversee the  system integrator and raised concerns about the agency’s ability to keep  costs under control in future program years by ensuring adequate  competition for Deepwater assets. Now that the Deepwater program is  underway, you asked us to assess whether the Coast Guard is effectively  managing the acquisition program and overseeing the contractor and to  assess the implications of using the Deepwater contracting model on  opportunities for competition.  Specifically, we assessed whether the  Coast Guard (1) has a structure in place to effectively manage the program  and oversee the system integrator, (2) has established adequate criteria by  which to assess and reward the system integrator’s performance, (3) has  the information needed to make a decision about extending the contract  after the first 5 years, and (4) is addressing the role and extent of  competition for Deepwater assets.
subsections: section_title: Background
paragraphs: Deepwater is the largest and most complex procurement project in the  Coast Guard’s history. The acquisition is scheduled to occur over a 30-year  period at a projected cost of $17 billion. It includes the modernization and  replacement of an aging fleet of over 90 cutters and 200 aircraft used for  missions that generally occur beyond 50 miles from shore. Deepwater  currently accounts for almost one-third of the Coast Guard’s acquisition  staff and one-third of support personnel funding.
Rather than using the traditional approach of replacing classes of ships or  aircraft through a series of individual acquisitions, the Coast Guard chose  to employ a “system-of-systems” acquisition strategy that would replace its  aging deepwater assets with a single, integrated package of new or  modernized assets. The primary objectives of the Deepwater program are  to maximize operational effectiveness and to minimize TOC while  satisfying the needs of the customer—the operational commanders,  aircraft pilots, cutter crews, maintenance personnel, and others who will  use the assets.
The Deepwater program has been in development for a number of years.  Between 1998 and 2001, three industry teams competed to identify and  provide the assets—aircraft, helicopters, cutters, logistics, and   C4ISR—needed to transform the Coast Guard. In June 2002, the Coast  Guard awarded a contract to ICGS as the system integrator for Deepwater.  Under a 5-year base contract with five additional 5-year options, ICGS is  responsible for designing, constructing, deploying, supporting, and  integrating the Deepwater assets to meet Coast Guard requirements. ICGS,  a business entity comprised of a nine-member Board of Directors, is jointly  owned by Northrop Grumman and Lockheed Martin. Northrop Grumman  and Lockheed Martin are the two first-tier subcontractors for the  Deepwater program. They, in turn, provide Deepwater assets themselves  or award second-tier subcontracts for the assets.
In our 2001 report, we identified several areas of risk for Deepwater. First,  the Coast Guard faced potential risk in the overall management and day- to-day administration of the contract. We reported on the major challenges  in developing and implementing plans for establishing effective human  capital practices, having key management and oversight processes and  procedures in place, and tracking data to measure system integrator  performance. In addition, we, as well as the Office of Management and  Budget, expressed concerns about the potential lack of competition during  the program’s later years and the reliance on a single system integrator for  procuring the Deepwater equipment. We also reported there was little  evidence that the Coast Guard had analyzed whether the approach carried  any inherent risks for ensuring the best value to the government and, if so,  what to do about them.
Since fiscal year 2002, Congress has appropriated almost $1.5 billion for  the Deepwater program (see table 1), and as of September 2003, the Coast  Guard had obligated $596 million to ICGS and $120.4 million for  government program management, legacy asset sustainment, and facilities  design work.
In response to congressional direction to assess the feasibility of  accelerating the Deepwater program, the Coast Guard reported in March  2003 that it could accelerate the implementation schedule from 20 years to  10 years and that this acceleration would provide increased operational  capability sooner to support maritime homeland security.
On March 1, 2003, the Coast Guard became part of DHS. A December 22,  2003, acquisition decision memorandum from the DHS Investment Review  Board’s (the Board) acting chairperson to the Coast Guard Commandant  stated that the Deepwater program has been designated as a level 1  investment, meaning that it will be reviewed at the highest levels within  the department. Further, because Deepwater interoperability within DHS  and with the Department of Defense will be a major program challenge,  the DHS Joint Requirements Council will be kept informed of Deepwater  developments. While decisions as to specific assets or capabilities have  been deferred to the Coast Guard acquisition executive, the Board will  meet to discuss actual or projected changes to the program that affect  cost, schedule, or performance. Noting that the Coast Guard has proposed  accelerating the Deepwater program in fiscal year 2005, the Board directed  the Coast Guard to ensure that risk management planning receives  appropriate attention, that TOC is kept current, and that cost, schedule,  and performance are monitored by measuring actual data against the  baseline and projections to completion.
section_title: Key Components of Management and Oversight Have Not Been Effectively Implemented
paragraphs: Complex, performance-based contracts such as Deepwater require  effective government oversight to ensure that the intended results are  achieved and that taxpayer dollars are not wasted. Both Coast Guard and  ICGS officials have acknowledged that an unusually large degree of  collaboration and partnership between the government and the system  integrator must be in place for the Deepwater acquisition to be successful.  However, a year and a half into the program, the key management and  oversight components needed to make the program effective have not  been effectively implemented. Integrated product teams (IPT) are the  Coast Guard’s primary tool for managing the program and overseeing the  contractor, but these teams have struggled to collaborate effectively and  accomplish their missions. While the Coast Guard has a Deepwater human  capital plan in place to guide strategic planning for turnover among  Deepwater personnel, the plan is not being followed and vacancies exist in  key positions. Further, while it is still early in the program, the transition  from existing Coast Guard assets to the new Deepwater assets has not  been effectively communicated, a particular concern in light of schedule  delays for some of the first assets to be delivered. Finally, a number of  plans integral to the organization and management of the Deepwater  program were finalized much later than anticipated. Appendix II contains  additional information on these plans.
subsections: section_title: IPTs Have Had Difficulty Fulfilling Their Critical Management Function
paragraphs: IPTs are the Coast Guard’s primary tool for managing the Deepwater  program and overseeing the system integrator. More than 30 of these  teams, comprised of Coast Guard, ICGS, and subcontractor employees  from Lockheed Martin and Northrop Grumman, are responsible for overall  program planning and management, asset integration, and overseeing the  delivery of specific Deepwater assets. However, the teams have struggled  to effectively carry out their missions. Our prior work at the Department  of Defense has shown that effective IPTs have (1) expertise to master  different facets of product development, (2) responsibility for day-to-day  decisions and product delivery, (3) key members who are either physically  collocated or connected through virtual means to facilitate team cohesion  and the ability to share information, and (4) control over their  membership, with membership changes driven by each team’s need for  different knowledge.
The Deepwater program manager reported IPT performance shortcomings  as an issue in 14 of the 16 monthly program assessment reports provided  to us. The following comments, made in Deepwater management reports  by Coast Guard officials involved on a number of different IPTs, convey  the difficulties faced by the teams in the first year and a half of the  program. Though the comments in table 2 are not exhaustive, they  demonstrate that the Deepwater IPTs have not been effective.
Based on our review of program reports, we identified four major issues  that have impeded the effective performance of the IPTs.
Lack of timely charters to vest IPTs with authority for decision  making. Authority for day-to-day decisions—required for program  success in meeting cost, schedule, and performance objectives—is  vested in IPTs through charters; yet charters for most of the Deepwater  IPTs were not developed in a timely manner. In fact, 27 of the 31 IPT  charters were not approved until after the first year of the contract.  More than merely a paperwork exercise, sound IPT charters are critical  because they detail each team’s purpose, membership, performance  goals, authority, responsibility, accountability, and relationships with  other groups, resources, and schedules. Between June 2002 and June  2003, 20 delivery task orders, authorized for issuance by the  contracting officer, were executed by IPTs that did not have charters in  place. Similarly, we found that some sub-IPTs, which address specific  issues at a subasset or component level, were operating on an ad hoc  basis without charters. For example, a November 2002 management  report states that sub-IPTs addressed numerous issues concerning  requirements for the national security cutter, even though their  charters were not approved until a year later. In addition, two other  sub-IPTs were not chartered.
Inadequate communication among members. The Coast Guard’s  Deepwater program management plan has identified collocation of IPT  members as a key program success factor, along with effective  communications within and among teams. Face-to-face informal  communication enhances information flow, cohesion, and  understanding of other members’ roles—all of which help foster team  unity and performance. Yet only 3 of the 31 operating IPTs are entirely  collocated, meaning that every IPT member is in the same building. The  IPTs responsible for assets frequently have members in multiple  locations. For example, the logistics process and policy development  IPT has members in 6 different locations. As noted in table 2, Coast  Guard IPT members have raised geographic separation as an issue of  concern. ICGS developed a Web-based system for government and  contractor employees to regularly access and update technical  information, training materials, and other program information, in part  to mitigate the challenges of having team members in multiple  locations. However, Coast Guard documents indicate that the system is  not being updated or used effectively by IPTs. In fact, the Deepwater  program executive officer reported that, while the system has great  potential, it is a long way from becoming the virtual enterprise and  collaborative environment required by the contractor’s statement of  work.
High turnover of IPT membership and understaffing. Most of the  Deepwater IPTs have experienced membership turnover and staffing  difficulties, resulting in a loss of team knowledge, overbooked  schedules, and crisis management. In a few instances, such as the  national security cutter and maritime patrol aircraft, even the IPT  leadership has changed. Also, key system integrator officials serving on  the management IPTs have left the company. Both the Chief Financial  Officer and the President of ICGS left their positions during the latter  half of 2003, and an additional six of the nine ICGS Board members  have changed. In addition, Coast Guard and system integrator  representatives have also been staffed on multiple IPTs, and, in many  cases meetings were attended by fewer than 50 percent of IPT  members. A December 2002 Coast Guard document summarizing  various programmatic recommendations cited a contractor study that  recommended individuals be assigned to IPTs on a full-time basis and  that they not serve on more than two teams. However, as of December  2003, 15 individuals were serving on three or more IPTs.
Insufficient training. The system integrator has had difficulty  training IPT members in time to ensure that they could effectively carry  out their duties, and program officials have referred to IPT training as  deficient. IPT charters state that members must complete initial  training before beginning team deliberations regarding execution of  new contracts for Deepwater assets. IPT training is to address, among  other issues, developing team goals and objectives, key processes, use  of the Web-based system, and team rules of behavior. According to a  Coast Guard evaluation report, IPT training was implemented late,  which has contributed to a lack of effective collaboration among team  members.
The Coast Guard hired a consultant to survey IPT members concerning  teams’ performance from July 2002 to September 2003. Three surveys  consisted of questions about mission, team member cooperation,  performance, communication, and integrated product and process  development. The final report on the survey results highlighted the need  for improved communications both within and among teams. Respondents  were also concerned that workloads were too high.
section_title: Human Capital Issues Pose a Challenge
paragraphs: In our 2001 report, we noted that as the Deepwater program got off the  ground, tough human capital challenges would need to be addressed. A  critical challenge we raised was the need to recruit and train enough staff  to manage and oversee the contract. To date, the Coast Guard has not  funded the number of staff requested by the Deepwater program and has  not adhered to the processes outlined in its human capital plan for  addressing turnover of Deepwater officials. These staffing shortfalls have  contributed to the problems IPT members have identified—such as the  struggle to keep pace with the workload and the difficulties in making  decisions due to inconsistent attendance at IPT meetings.
Although the Deepwater program has identified the need for a total of 264  staff in fiscal year 2004, only 224 positions have been funded, and only 209  have been assigned to the program. The Coast Guard’s fiscal year 2004  funding for personnel was increased to $70 million; however, the Coast  Guard did not request sufficient funds to fill the 40 positions that the  Deepwater program identified as necessary. According to Coast Guard  officials, $70 million is insufficient to fund their fiscal year 2004 personnel  plan because they need $67 million of this amount just to fund current  personnel levels. The assistant commandant has imposed a temporary  hiring freeze and plans to monitor expenditures throughout the year to  identify any available funding for additional positions. Although we asked,  Coast Guard officials did not explain why they did not request sufficient  funds to adequately staff the program.
In addition, the Coast Guard has not adequately addressed the imminent  departure of Coast Guard officials from the Deepwater program. Coast  Guard officials will leave each year due to the normal rotational cycle of  military members (every 3 to 4 years) and retirements. The Deepwater  human capital plan sets a goal of a 95 percent or higher “fill rate” annually  for both military and civilian positions and proposes using a “floating”  training position that can be filled by replacement personnel reporting for  duty a year before the departure of the military incumbents. This position  is meant to ensure that incoming personnel receive acquisition training  and on-the-job training with experienced Deepwater personnel. However,  the 2004 request for this training position was not funded, nor was funding  provided for additional new positions identified as critical. In December  2003, the Director of Resources and Metrics and the Chief Contracting  Officer left the Deepwater program, and the program manager is slated to  leave in March 2004. In addition, by July 1, 2004, five key Coast Guard  officials who oversee the work of the asset IPTs are scheduled to leave.   Coast Guard officials told us that they have identified the military  replacements that will join the program in the summer of 2004.
section_title: Transition Planning for New Assets Needs Attention
paragraphs: Although Deepwater is still in the early stages, assets will start to be  delivered incrementally to operating units soon. The first Deepwater  assets—the 123-foot cutter and short range prosecutor—are scheduled to  be delivered to operating divisions in 2004.  Operating units will receive  additional ships, aircraft, or C4ISR every year thereafter until the  Deepwater program ends. However, the Coast Guard has not  communicated decisions on how the new and old assets are to be  integrated during the transition and whether Coast Guard or contractor  personnel—or both—will be responsible for maintenance. Coast Guard  field personnel, including senior-level operators and naval engineering  support command officials, told us that they have not received information  about how they will be able to continue meeting their missions using  current assets while being trained on the new assets. They are also unclear  as to whether the system integrator or Coast Guard personnel will be  responsible for maintenance of the new assets. For example, although  Deepwater officials have stated that maintenance on the new assets will  be a joint responsibility, naval engineering support command staff had  received no instruction on how this joint responsibility is to be carried out.  Coast Guard officials told us that guidance on joint maintenance  responsibility has not been completely disseminated throughout the Coast  Guard, but said that ICGS has recently added representatives at the key  maintenance and logistics sites to coordinate maintenance issues.
One of the first Deepwater assets to be delivered is the 123-foot cutter. The  Coast Guard is modifying its 110-foot cutter by adding 13 feet of deck and  hull, a stern ramp, a superstructure, and communication equipment. The  123-foot cutter is an example of the transition challenges facing the Coast  Guard. First, there is confusion over which of the cutters will be modified  and when. The contract with ICGS calls for all 49 cutters to be modified;  however, Deepwater officials are considering curtailing the modification  efforts and accelerating the development of the fast response cutter  instead. (The fast response cutter was originally planned to be delivered in  2018 as a replacement for the 123-foot cutter). In addition, the Coast Guard  identified 22 of the 110-foot cutters that, due to unexpectedly severe hull  corrosion, required additional inspection and repair separate from the  Deepwater modification plans. To date, $14.7 million in non-Deepwater  funds has been made available to repair 8 of these cutters. Further, Coast  Guard officials note that there are 4 cutters in operation in the Persian  Gulf, which makes them unavailable for modification at this time.  System  integrator and Coast Guard officials expressed confusion about the status  of the cutter modifications, hull repair program, and fast response cutter  schedule. For example, ICGS officials indicated that they did not know  what the Coast Guard plans for the 123-foot cutter modification. The Coast  Guard is considering several options and has not made a final decision on  the cutter modification effort.
Transitioning the staffing and operations of current Coast Guard assets to  Deepwater assets may be further complicated by schedule delays. Reliable  information on the delivery of Deepwater assets is important to the  planning and budgeting efforts of Coast Guard operators and maintenance  personnel to ensure that current missions are met and existing assets are  maintained. Delivery of the first 123-foot cutter and short range prosecutor  is scheduled for March 2004, slipping from the original delivery date of  November 2003, and the rescheduled date of December 2003. This delay is  affecting the schedules for the remaining cutters under contract, according  to the most recent program manager assessment. Program management  reports also indicate that schedule milestones have slipped for the  maritime patrol aircraft.  The first two aircraft are currently scheduled to  be delivered to operating divisions in late 2006 or early 2007, compared  with the original plan of 2005. The IPT is working toward design of the  aircraft, even though Coast Guard approval to proceed has not been set  forth in the form of a definitized contract for this asset.  The target date  for definitizing the contract is now April 2004.
section_title: Process for Assessing System Integrator’s Ongoing Performance Lacked Rigor
paragraphs: According to Office of Federal Procurement Policy guidance, a  performance-based contract such as Deepwater should have measurable  performance standards and incentives to motivate contractor  performance. Contractors should be rewarded for good performance  based on measurement against predetermined performance standards. In  general, the contractor is to meet the government’s performance  objectives, at appropriate performance quality levels, and be rewarded for  outstanding work. Further, sound internal controls are important to  ensure that plans, methods, and procedures are in place to support  performance-based management. Relevant information should be  recorded and communicated to management and others in a form and a  time frame that enables them to carry out their responsibilities.
The Coast Guard’s process and procedures for evaluating the system  integrator’s performance during the first year of the contract lacked rigor  in terms of applying quantifiable metrics to assess performance, gathering  input from government performance monitors, and communicating with  and documenting information for the decision makers. The process used  to hold the system integrator accountable for results was not transparent  and, in fact, contained several inconsistencies that raise questions as to  whether the Coast Guard’s decision to give the contractor 87 percent of  the award fee was based on accurate information.
subsections: section_title: Quantifiable Award Fee Metrics Are Still Being Developed
paragraphs: The Coast Guard measures the system integrator’s ongoing performance  based on periodic assessments using weighted evaluation factors. The  award fee for the first year of performance of the overall integration and  management of the Deepwater program was based on an evaluation of the  following five factors:    overall program management,    cost monitoring and control,    quality,  innovation, and  flexibility.
These evaluation factors are further defined in the contract’s award fee  plan. For example, innovation is the “extent to which innovation, designs,  processes, and concepts have been introduced that result in operational  performance improvements and/or total ownership cost reductions.”  While there will inevitably be a degree of subjectivity in award fee  decisions, the Coast Guard lacks quantifiable metrics to make an  assessment of the contractor’s performance.  Given the lack of specificity  of the metrics, it is not clear how they could be used to make such an  assessment, particularly on a program as complex as Deepwater. Coast  Guard officials acknowledged that the factors need to be better-defined,  with supporting metrics that would provide a more objective basis for  future award fee assessments.
In the meantime, a May 31, 2003, Coast Guard memorandum to ICGS  indicates that the contractor will be rated based on three factors for the  second year of performance rather than five. However, the factors are  vague and undefined: quality, program management, and system  engineering. Further, supporting metrics to measure these performance  factors have not been developed.
section_title: Information Was Not Gathered and Communicated Accurately
paragraphs: Under the Deepwater contract’s award fee plan and the program  management plan, technical specialists, known as contracting officers’  technical representatives (COTR), are to provide their observations to a  program evaluation board comprised of the contracting officer, the  program manager, and two COTRs. The Deepwater program executive  officer then makes the final award fee determination based on the board’s  recommendations. The Coast Guard’s award fee evaluation of the first  year of ICGS’s performance was based on unsupported calculations and  relied heavily on subjective considerations. As a result, the basis for the  final decision to provide the contractor an award fee rating of 87 percent,  which falls in the “very good” range, was not well-supported.
For example, while all COTRs submitted comments, the assessment did  not include numerical and adjectival ratings from all COTRs. Input from  the COTR responsible for gauging the system integrator’s performance for  all efforts related to the design and delivery of ships was not included in  the calculation at all. Prior to speaking with us, the COTR did not know his  input was absent from final performance monitor calculations, which  resulted in a recommended rating of 82.5 percent. Input from two other  COTRs was provided for some but not all of the five evaluation factors. A  fourth COTR provided only adjectival ratings, whereas others provided  numerical scores. Subsequently, and unbeknownst to the COTR, a  program evaluation board member calculated a numerical score for this  COTR’s observations. While an adjectival rating of “good,” for example,  could range from a score of 71 to 80, the board member scored each of the  factors in the midrange. Scoring this COTR’s adjectival ratings in the low  or high end of the range would have produced a different outcome.  Program evaluation board officials were not aware of the inconsistencies  in the calculations until we informed them.
One program evaluation board member raised concerns that the board’s  subsequent award fee recommendation of 90 percent was too high and  that the assessment focused disproportionately on the system integrator’s  performance in the last part of the year rather than its performance over  the entire year. In addition, the program manager’s assessment stated that  “overall program management…needs substantial improvement.” Further,  Coast Guard management reports throughout the first year of the contract  cited various schedule, performance, cost control, and contract  administration problems that required attention. Among the assets cited as  needing attention were the maritime patrol aircraft, the short range  prosecutor, the 123-foot cutter, and the logistics integration management  system. Ultimately, the program executive officer awarded the system  integrator a rating of 87 percent, resulting in an award fee of $4.0 million  of the maximum $4.6 million annual award fee.
Coast Guard officials told us that they will now assess ICGS’s performance  every 6 months, rather than annually. However, the contract has not been  modified to reflect either the changes to the evaluation factors, discussed  previously, or the new assessment period. The first 6-month assessment  was scheduled for completion in December 2003, but as of March 2004,  Coast Guard officials told us it is currently ongoing.
The contractor was eligible for a second award fee of up to $1.5 million in  August 2003 for performance related to the continuous improvement of  elements common to C4ISR and life cycle and logistics engineering for all  assets.  Coast Guard officials said that they awarded the system integrator  79 percent of the maximum award fee; however, they did not provide us  with supporting documentation of the award fee determination process.
section_title: Coast Guard Has Not Begun to Measure Contractor Performance against Deepwater Program Goals
paragraphs: The Coast Guard is scheduled to decide on extending ICGS’s contract by  June 2006, 1 year prior to the end of the first 5-year contract term. In 2001,  the Coast Guard set a goal of developing measures, within a year after  contract award, to conduct annual assessments of the system integrator’s  progress toward achieving the three overarching goals of the Deepwater  program: increased operational effectiveness, lower TOC, and customer  satisfaction. However, the Coast Guard’s time frame for implementing  metrics to gauge progress against these goals has slipped. Further, the  baseline the Coast Guard is using to assess TOC will not provide the  government with critical information it needs about the efficiencies of  using the Deepwater approach. Therefore, the Coast Guard is not in a  position to begin the decision-making process about whether or not to  extend the contract past the 5-year base period.
The time frame for the first review of the contractor’s performance against  the Deepwater goals has slipped. It was originally rescheduled for 18  months after contract award (December 2003), 6 months later than  planned.  Deepwater officials told us that the performance review is  currently ongoing and is expected to be completed in March 2004.  While  the Coast Guard has begun to develop models to measure the extent to  which Deepwater is achieving increased operational effectiveness and  reduced TOC, a decision has not yet been made as to which specific suite  of models will be used.  The former Deepwater chief contracting officer  told us he anticipates that the metrics will be in place in year 4 of the  contract, the same year the decision needs to be made to extend the  contract. Other officials acknowledged that it is difficult to hold the  contractor accountable for progress toward the goals this early in the  program, but could not offer a projection as to when the operational  effectiveness and TOC results would be forthcoming.
Coast Guard officials noted the large degree of complexity involved in  attempting to measure the system integrator’s progress toward the  Deepwater goals. In previous work, we found that assessing improvements  in operational effectiveness and TOC may be difficult because  performance data may reflect factors that did not result from the  contractor’s actions. Because the Deepwater program includes legacy  assets, modified assets, and new assets, the line of accountability between  the government and the system integrator is blurred. It is not always clear  who is responsible between Coast Guard and ICGS for the change in  performance or costs of Deepwater assets. Measuring the 123-foot cutter’s  performance, for example, is complicated by the fact that ICGS is  responsible for the new 13 feet of deck and hull and other modifications,  while the engine and the other 110 feet of the deck and hull are the Coast  Guard’s responsibility.
Coast Guard officials said that they are measuring “operational  performance,” such as the number of search and rescue, drug interdiction,  and migrant interdiction missions carried out by the current assets.  However, they could not explain how these measures will be used to  assess ICGS’s progress toward improving operational effectiveness with  Deepwater assets. The officials stated that the models they are using to  measure operational performance for the various Coast Guard missions  lack the fidelity to capture whether improvements may be due to Coast  Guard or contractor actions, the capability of specific Deepwater assets,  or even outside factors such as improved intelligence on drug smugglers.  Program officials noted that it is difficult to hold the contractor  accountable for operational effectiveness at this point, before Deepwater  assets are delivered.
Establishing a solid baseline against which to measure progress in  lowering TOC is critical to holding the contractor accountable. However,  the Coast Guard is using as the baseline ICGS’s own projected cost of  $70.97 billion plus 10 percent (in fiscal year 2002 dollars).  Therefore, the  government will not have the TOC information it needs to make a contract  extension decision. Measurement of ICGS’s cost as compared to its own  cost proposal will tell the Coast Guard nothing about the efficiencies it  may be getting using the Deepwater performance-based approach.  Further, the baseline the Coast Guard is using has been significantly  changed from that originally envisioned. The Deepwater program  management plan, approved in December 2003, states that the estimated  cost to replace individual Coast Guard assets under a traditional approach,  (i.e., without the ICGS Deepwater “system of systems” solution), is to be  the “upper limit for TOC” that the contractor should not exceed. The  officials could not explain why the program management plan, which sets  forth the overall framework for managing Deepwater, contains a different  TOC baseline than the one they are using.
Further, changes in such variables as fuel costs or cutters’ operating  tempo could result in additional changes being made to the TOC baseline.  Coast Guard officials explained that proposed changes to the baseline  would be approved by the program executive officer on a case-by-case  basis. However, the Coast Guard has not developed criteria for potential  upward or downward adjustments to the baseline.
The Coast Guard has only recently begun to address the contractor’s  progress in meeting the third overall goal of Deepwater, customer  satisfaction. A January 9, 2004, report indicates that the Coast Guard had  not yet identified the metrics needed to measure this goal. As a start, on  January 12, 2004, a survey was sent to 25 senior leaders and program  managers.
The Coast Guard decided to use the system integrator approach 6 years  ago. In our previous work, we found that given the Coast Guard’s reliance  on a single system integrator for the Deepwater program, the agency  would be at serious risk if it decides not to extend the contract.  Because  ICGS proposed the specific assets that became the Deepwater solution, a  decision not to extend the current contract would require a new  Deepwater acquisition strategy to be developed. Exit strategies and other  means to deal with potential poor performance by the system integrator  are important to mitigate these program risks. However, the Coast Guard  is just beginning an internal review of the system integrator’s plan to  transition out of the program in the event such action would be necessary.  Further, Deepwater program officials indicated that it is not realistic to  believe the Coast Guard would switch system integrators at this point in  the program. They stated that they viewed their relationship with the  contractor as a partnership and are committed to making it work.
section_title: Control of Future Costs through Competition Remains a Risk because of Weak Oversight of Subcontractor Decisions
paragraphs: Competition is a key component for controlling costs in the Deepwater  program and a guiding principle for DHS’s major acquisitions. The benefits  of competition may be viewed as sufficient in the contract’s early years  because, for the initial 5-year contract period, prices proposed by ICGS for  equipment and software were based on competitions held among various  subcontractors. However, beyond the first 5-year term, the Coast Guard  has no way to ensure competition is occurring because it does not have  mechanisms in place to measure the extent of competition or to hold the  system integrator accountable for steps taken to achieve competition.
The acquisition structure of the Deepwater program is such that the two  first-tier subcontractors, Lockheed Martin and Northrop Grumman—the  companies that formed ICGS and that developed the Deepwater   solution—have sole responsibility for determining whether to hold  competitions for Deepwater assets or to provide these assets themselves.  Over 40 percent of the funds obligated to Lockheed Martin and Northrop  Grumman have either remained with those companies or been awarded to  their subsidiaries. Further, the system integrator uses a Lockheed Martin  sourcing document, termed the open business model, to guide competition  decisions made by the subcontractors. However, this guidance is a  philosophy—not a formal process involving specific actions—that  encourages competition but does not require it. The lack of transparency  into competition and the government’s lack of a mechanism to hold the  contractor accountable raise questions about whether the Coast Guard  will be able to control costs.
subsections: section_title: Coast Guard Lacks Visibility into Subcontractors’ Make or Buy Decisions
paragraphs: Neither the Coast Guard nor the system integrator determines how  suppliers for Deepwater assets are chosen.  A Coast Guard official told us  that the system integrator was hired to make these decisions because the  agency lacked the expertise to do so. However, Lockheed Martin and  Northrop Grumman, as the subcontractors, are solely responsible for  deciding whether to hold competitions for Deepwater assets or provide  them to the Coast Guard themselves (often referred to as “make or buy”  decisions).
Moreover, the Coast Guard has no contractual requirements with ICGS  that provide transparency into significant make or buy decisions.   Although the Coast Guard has decided to include achieving competition as  one of the factors to be considered in decisions about extending the  contract for future option terms, this review will occur after such  subcontracting decisions are made. The subcontractors are not required to  notify the Coast Guard prior to making a decision to provide Deepwater  assets themselves rather than holding a competition.  The Coast Guard’s  review of competition included in its award term plan will not address  Lockheed Martin or Northrop Grumman decisions—increasingly  important in subsequent years—of whether significant equipment should  be procured from outside sources or built in-house.
As of September 30, 2003, the Coast Guard had awarded $596 million in  orders to the system integrator, ICGS. Table 3 shows that over 98 percent  of this amount was then passed through to the two first-tier  subcontractors.
To date, the subcontractors managing the acquisition have frequently  performed the work themselves. Based on their respective work scopes,  the two companies either issue orders to second-tier subcontractors or  retain the work for themselves. Table 4 shows that, as of September 30,  2003, Lockheed Martin planned to retain 42 percent of its obligated dollars  and to award 58 percent to second-tier subcontractors. Most of these  second-tier dollars will go to major subcontractors, i.e., those with  obligations greater than $5 million.
As shown in table 5, Northrop Grumman planned to retain 51 percent of its  obligated dollars.
section_title: The System Integrator’s Open Business Model Does Not Ensure Competition Is Considered
paragraphs: The open business model, meant to guide the supplier sourcing process in  the Deepwater program, has been characterized by the system integrator  as a means of ensuring competition for Deepwater assets throughout the  life of the program, thereby keeping costs under control. In October 2003,  ICGS issued a policy statement on the open business model. The stated  business approach of the guidance is to encourage second-tier suppliers to  remain innovative and competitive by directing Lockheed Martin and  Northrop Grumman, as the first-tier subcontractors, to (1) generally avoid  the use of teaming agreements with suppliers and prohibit teaming  agreements based on guaranteed work share, (2) defer second-tier  supplier decisions as long as practicable so that changes in the  marketplace can be considered, and (3) actively solicit market information  and new suppliers. However, this guidance is a philosophy—not a formal  process involving specific decision points—that does not ensure that  competition will be considered. The December 2003 Deepwater  performance measurement plan requests that the contractor prepare self- assessments of its efforts to promote competition. However, the Coast  Guard has no means of obtaining insight into the basis for the contractor’s  self-assessments. Moreover, the government still lacks a mechanism to  hold the contractor accountable for ensuring that competition occurs.
To date, there have been varying degrees of competition for the second- tier subcontracting relationships Lockheed Martin and Northrop Grumman  have in place for the design, development, or production of Deepwater  assets. Lockheed Martin and Northrop Grumman follow their own  procurement procedures and guidance for determining whether  competition will occur and selecting the suppliers who will be invited to  compete for Deepwater assets.  The competitions are not “full and open”  in the way a typical government procurement would be, nor are they  required to be.  The federal procurement system requires “full and open”  competition except in cases where certain statutory exceptions are met.   “Full and open” competition means that all responsible sources are  permitted to compete.
ICGS officials identified four specific assets for which they believe the  open business model philosophy was effective: the conversion of 110-foot  to 123-foot cutters, the national security cutter, the maritime patrol  aircraft, and the vertical take-off and landing unmanned aerial vehicle  (VUAV). We found that, in some cases, teaming agreements were  implemented in the proposal phase of Deepwater and were carried over  when ICGS won the contract. In other cases, some degree of competition  had occurred.
In December 1998, Lockheed Martin Corporation, Ingalls Shipbuilding,  Inc., and Halter-Bollinger Joint Venture, L.L.C entered into a teaming  agreement that included the 123-foot cutter modification. The  agreement was established to make the capabilities of both Halter and  Bollinger available to Lockheed Martin for all phases of the Deepwater  program. Despite the open business model’s prohibition of work share  agreements, such an agreement is in place between Lockheed Martin  and Halter-Bollinger. Halter-Bollinger will be responsible for the design  and construction of all vessels equal to or less than 200 feet in overall  length, with the exception of the national security cutter. For those  ships greater than 200 feet and less than 241 feet, the company’s work  share is 25 percent of the total effort.
The national security cutters are being designed and constructed by  Northrop Grumman. Northrop Grumman awarded a contract to M.  Rosenblatt & Son, Inc. for the cutters’ preliminary design, but Northrop  Grumman is responsible for the detailed design and construction.  Lockheed Martin is responsible for the electronics. However, Northrop  Grumman plans to hold competitions for the long-lead materials—such  as the gas turbine, bulkhead seals, stern tubes, and rudder—for the  cutters and has solicited pricing proposals from a number of  subcontractors.
Prior to the contract award to ICGS, Lockheed Martin solicited  information from a number of companies for the maritime patrol  aircraft, evaluating 16 aircraft proposals. In October 2000, Lockheed  Martin signed a memorandum of understanding with CASA Aircraft  USA, Inc. to provide an airframe for Deepwater and to help develop  and market ICGS’s Deepwater proposal. After some Coast Guard  officials expressed concern about the aircraft model that had been  selected for Deepwater, ICGS was awarded a task order to pay for an  evaluation of alternative aircraft. As a result of the evaluation,  Lockheed Martin identified an alternative CASA aircraft to meet the  Coast Guard’s maritime patrol aircraft mission.
For the VUAV, Lockheed Martin conducted a competition between six  models. Bell Helicopter, Inc. was initially identified as the solution.   After ICGS submitted its Deepwater proposal to the Coast Guard,  Lockheed Martin identified a potential Northrop Grumman product  based on market research. However, after evaluation of this alternative,  Lockheed Martin selected one of the Bell products.
section_title: Conclusion
paragraphs: The Coast Guard has embarked on a major transformational effort using  an acquisition strategy that allows a system integrator to identify the  Deepwater assets and to manage the acquisition process, with  subcontractors retaining authority for all make or buy decisions.  Such a  strategy carries inherent risks that must be mitigated by effective  government oversight of the contractor. The Coast Guard faces a tough  challenge in holding ICGS accountable for results, while facing the  daunting prospect of starting over with a new approach should the  contractor fail. Nevertheless, the integrity of the contractor oversight  process must be enforced through such mechanisms as effective IPTs and  a rigorous and transparent award fee determination process.
Further, the Coast Guard must determine how to hold the contractor  accountable for achieving the basic goals of the Deepwater program in  order to position itself to make a contract extension decision. While there  is no question that the success of Deepwater depends on an effective  partnership between the government and the contractor, the Coast Guard  must preserve its ability to change course if necessary. Solid baselines  need to be developed so actual costs and operational effectiveness of the  Deepwater assets can be accurately measured and reported. The current  use of the contractor’s proposed costs, plus 10 percent, as the TOC  baseline—rather than the estimated cost to replace the assets via a  traditional procurement approach—is troublesome. Further, because the  program management plan does not reflect the change to the TOC  baseline, we question whether this decision was well thought-out and in  the government’s best interest.
In addition to contractor oversight, the Coast Guard has not invested the  resources needed to ensure that its own personnel are trained and staffed  in sufficient numbers to carry out their duties. The disconnect between the  process outlined in the human capital plan for ensuring a smooth  transition as military personnel rotate out of Deepwater and the current  situation—where key Deepwater officials are leaving the program without  a chance to adequately train their replacements—is cause for concern as  the Deepwater program moves forward. It is unclear why the Coast Guard  has not devoted adequate attention to human capital needs. In addition,  although the first Deepwater assets are just starting to be delivered, the  lack of a solid and well-developed transition plan from legacy to  Deepwater assets is already causing problems, as evidenced by the 123- foot cutter modification difficulties. The schedule delays for several of the  assets further highlight the need for more focus on the transition to  Deepwater assets.
The concerns we raised in 2001 about the Coast Guard’s ability to control  costs in future years remain valid today. Without a mechanism to hold the  system integrator accountable for ensuring adequate competition, the  Coast Guard cannot be sure that competition will be used to guard against  cost increases that could jeopardize the program. This situation is  especially risky given the acquisition structure of Deepwater, whereby the  subcontractors, not the system integrator or the Coast Guard, are  responsible for determining whether competition will occur for Deepwater  assets.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Homeland Security direct the  Commandant of the Coast Guard to take the following three actions to  address Deepwater program management:  In collaboration with the system integrator, take the necessary steps to  make IPTs effective, including  training IPT members in a timely manner,    chartering the sub-IPTs, and    making improvements to the electronic information system that  would result in better information sharing among IPT members who  are geographically dispersed.
Follow the procedures outlined in the human capital plan to ensure  that adequate staffing is in place and turnover among Deepwater  personnel is proactively addressed.
As Deepwater assets begin to be delivered to operational units, ensure  that field operators and maintenance personnel are provided with  timely information and training on how the transition will occur and  how maintenance responsibilities are to be divided between system  integrator and Coast Guard personnel.
Further, we recommend that the Secretary direct the Commandant to take  the following six actions to improve contractor accountability:    Develop and adhere to measurable award fee criteria consistent with  the Office of Federal Procurement Policy’s guidance.
In all future award fee assessments, ensure that the input of  COTRs is  considered and set forth in a more rigorous manner.
Hold the system integrator accountable in future award fee  determinations for improving the effectiveness of IPTs.
Based on the current schedule for delivery of Deepwater assets,  establish a time frame for when the models and metrics will be in place  with the appropriate degree of fidelity to be able to measure the  contractor’s progress toward improving operational effectiveness.
Establish a TOC baseline that can be used to measure whether the  Deepwater acquisition approach is providing the government with  increased efficiencies compared to what it would have cost without  this approach.
Establish criteria to determine when the TOC baseline should be  adjusted and ensure that the reasons for any changes are documented.
To facilitate controlling future costs through competition, we also  recommend that the Secretary direct the Commandant to take the  following two actions:    Develop a comprehensive plan for holding the system integrator  accountable for ensuring an adequate degree of competition among  second-tier suppliers in future program years. This plan should include  metrics to measure outcomes and consideration of how these  outcomes will be taken into account in future award fee decisions.
For subcontracts over $5 million awarded by ICGS to Lockheed Martin  and Northrop Grumman, require Lockheed Martin and Northrop  Grumman to notify the Coast Guard of a decision to perform the work  themselves rather than contracting it out. The documentation should  include an evaluation of the alternatives considered.
section_title: Agency Comments
paragraphs: DHS forwarded us the Coast Guard’s written comments on a draft of this  report, which are reproduced in appendix I. The Coast Guard provided us  with additional technical comments, which we incorporated as  appropriate. In an e-mail sent subsequent to the written comments, the  Coast Guard stated that it agreed with our recommendations.
The Coast Guard noted that the agency is learning and evolving as the  Deepwater program matures and pointed out that many aspects of the  Deepwater program—working with a system integrator, employing IPTs  across multiple acquisition domains, and using a performance-based  strategy for such a long-term undertaking—are new to the Coast Guard.   The agency agreed that, because the IPT structure is new to the Coast  Guard, many adjustments must be made to improve the teams’  effectiveness. The Coast Guard clarified that the IPTs are, for the most  part, contractor-led and that Coast Guard IPT members provide support  and oversight.  The focus of this report, however, is on the government’s  ability to oversee and manage the contractor. Deepwater management  documents assert, as we point out in our report, that IPTs are the Coast  Guard’s primary tool for managing and overseeing the contractor.
Regarding the award fee process, the Coast Guard stated that it has taken  action to assimilate objective factors into future evaluations but expressed  concern that our draft report may not have completely reflected the rigor  that was applied in the first award fee decision.  The Coast Guard states  that “no input from any of the monitors was left out of the evaluation  process.”  While we revised our report to state that all COTRs submitted  comments, the input from the COTR responsible for ships was not  included in the numerical scores, which were then passed on to the fee  determining official.  Further, the Coast Guard said that the score of 87  percent is “much lower than industry averages.”  In our view, however, the  relevant consideration in determining the award fee amount is not  industry averages, but rather the purpose an award fee is intended to  serve.  The rationale for offering award fees is to motivate superior effort  on specific task and delivery orders, assets, or system performance  attributes. Of importance here, the narrative description in the Deepwater  award fee plan associated with a score of 87 percent (“very good”) is “very  effective performance, fully responsive to contract requirements . . . only  minor deficiencies.”  As we state in our report, program management  reports throughout the first year of the contract cited various schedule,  performance, cost control, and contract administration problems that  required attention.
The Coast Guard agreed that competition is critical to controlling costs  and indicated that it is planning efforts that will result in greater visibility  and increased accountability to ensure competitive practices are being  used to manage costs.
section_title: Scope and Methodology
paragraphs: To determine the steps taken by the Coast Guard to manage the  Deepwater program and oversee system integrator performance, we  examined the Deepwater contract, the program management plan, the  human capital plan, briefings, budget justifications, and monthly and  quarterly management reports. We analyzed IPT charters, membership  lists, survey data, and staffing data, and we observed IPT and working  group meetings. We interviewed various Deepwater program officials  representing the Coast Guard, the system integrator, and the  subcontractors, including program and asset-level program managers,  contracting officers, and ICGS representatives in Arlington, Virginia, and  Washington, D.C.. We visited the First and Seventh Coast Guard Districts  in Boston, Massachusetts, and Miami, Florida, and interviewed operators  and systems specialists for Coast Guard cutters, aircraft, and helicopters  at those locations. We also met with Lockheed Martin and Northrop  Grumman employees in Avondale, Louisiana, and Moorestown, New  Jersey. We reviewed our prior reports and testimonies on the Deepwater  project and integrated product teams.
To assess Coast Guard efforts to establish effective criteria to assess and  reward the system integrator’s performance after the first year of the  contract, we reviewed the award fee plan, the performance incentives  plan, the interim and final award fee reports for the first year of contract  performance, and other management documents. We interviewed Coast  Guard and ICGS officials. Our analysis of this issue was hindered by the  Coast Guard’s failure to provide us with two additional award fee  determinations, despite our repeated requests.
To assess whether the Coast Guard has put in place measures to assess  the contractor’s progress in meeting the three overarching goals of  Deepwater, we reviewed the performance measurement plan, the award  term plan, and other performance measurement documents. Additionally,  we interviewed the Deepwater program’s Resources and Metrics staff,  Coast Guard operations personnel, and program managers. We also  reviewed our prior report on performance-based contracting attributes.
To determine whether the Coast Guard is addressing the role and extent of  competition for Deepwater assets, we examined ICGS’s open business  model policy statement and excerpts from Lockheed Martin’s procurement  manual and Northrop Grumman’s acquisition policy manual. We discussed  the open business model with officials from the Coast Guard, ICGS,  Lockheed Martin, and Northrop Grumman.  In addition, we reviewed  financial data, including contract orders and ICGS spreadsheets. The  Coast Guard provided us with the obligations to ICGS, Lockheed Martin,  and Northrop Grumman. We did not independently verify the financial  data.
We performed our work from May 2003 through February 2004 in  accordance with generally accepted government auditing standards.
We are sending copies of this report to other interested congressional  committees, the Secretary of Homeland Security, and the Commandant of  the Coast Guard. We will make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-4841 or Michele Mackin, Assistant Director, at  (202) 512-4309. Other major contributors to this report were Penny  Berrier, Ramona L. Burton, Christopher Galvin, Lucia DeMaio, Gary  Middleton, and Ralph O. White Jr.
section_title: Appendix I: Comments from the Coast Guard
section_title: Appendix II: Status of Selected Deepwater Contract Management and Oversight Plans
subsections: section_title: Establishes criteria and procedures used to evaluate Integrated Coast Guard Systems’ (ICGS) management performance, and to determine the amount of award fee earned
paragraphs: Details responsibilities and processes to implement  the contract  Describes activities and processes to ensure  funding is available to execute the program  Establishes structure and method for identifying and  managing risks, and developing and selecting  options to mitigate risks  “Soon after  contract award”
Outlines the processes and procedures used to  implement the program’s quality assurance process  90 days after  contract award  (September 2002)