Defense Acquisitions: Success of Advanced SEAL Delivery System Hinges on Establishing a Sound Contracting Strategy and Performance Criteria
Published: 2007-05-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Advanced SEAL Delivery System (ASDS) is a hybrid combatant submersible providing clandestine delivery and extraction of Navy SEALs and equipment in high-threat environments. The first ASDS has had significant performance issues and has cost, to date, over $885 million. In May 2006, Congress requested that GAO review ASDS. This report examines (1) how the Navy managed ASDS risks through its contracts and (2) the status of major technical issues and program restructuring.
section_title: What GAO Found
paragraphs: The Navy did not effectively oversee the contracts to maintain, repair, and upgrade the ASDS and failed to hold the prime contractor accountable for results. The Navy took responsibility for correcting the boat's deficiencies while continuing to pay the costs and fees of the prime contractor under cost reimbursable contracts to execute the corrections. Before accepting the boat, the Navy went to sources other than the prime contractor to obtain better designs for the propeller and battery and then paid the prime contractor to install them. When the Navy accepted the ASDS in 2003 in an "as is" condition, it relieved the contractor from having to take any additional actions to correct known problems. Since then, the U.S. Special Operations Command has continued to invest millions of dollars to fix existing problems and address new ones in an attempt to make the boat operational. In making this additional investment, the Navy entered into contracts with the prime contractor that provided little incentive to control costs, authorized work before reaching agreement on the scope and price of the work to be performed, and failed to finalize the terms of the work within required time frames. Meanwhile, the contractor's performance continued to be poor, often exceeding initial estimates for the time and cost required to perform the work. ASDS officials took actions over the past 2 years to address these issues, but acknowledge that it is too early to determine the effectiveness of more recent actions to incentivize the contractor's performance. Continuing problems with the existing ASDS led to the Department of Defense's (DOD) April 2006 decision to cancel plans to buy additional ASDS boats, establish an improvement program for the in-service ASDS, and conduct an assessment of alternative material solutions to fulfill remaining operational requirements. The problems have seriously degraded the boat's reliability and performance, and the boat is only available for limited operational use. The results of these improvement and assessment efforts are expected to provide DOD the knowledge needed to determine whether ASDS's reliability can be improved cost-effectively to make ASDS an operational asset and whether an alternative development program is needed to meet the remaining operational requirements. A program decision is planned in mid-2008, after the ASDS improvement program and assessment of alternate material solutions are completed.

section_title: Letter
paragraphs: The Advanced SEAL Delivery System (ASDS), a hybrid combatant  submersible, is one of the U.S. Special Operations Command’s (SOCOM)  largest investments. The ASDS is designed for clandestine delivery and  extraction of Navy SEALs and equipment in high-threat environments. In  2006, the Department of Defense’s (DOD) Quadrennial Defense Review  Report revalidated the need for the type of capabilities that ASDS is  designed to provide.
The ASDS has encountered a difficult, long, and costly development since  the initial contract was awarded in 1994. The first ASDS boat did not meet  all technical or performance requirements, yet in 2003 it was accepted by  the Navy for operational use. Since acceptance, ASDS has exhibited  significant reliability and performance issues during test and operation,  and an in-service improvement effort has begun. For the most part, the  Navy has used the same contractor to design and deliver the boat, to  develop corrections to performance and reliability problems, and to  support the boat in the field. Total program costs—including research and  development, procurement, military construction, operations and  maintenance, and military personnel costs—have reached about   $885 million.
In May 2006, you requested that GAO review the status and problems  facing the ASDS program. We subsequently agreed to address the issues in  two separate efforts. This report examines how the Navy managed ASDS  risks through its contracts and the status of major technical issues and  program restructuring.
subsections: section_title: Background
paragraphs: The special operations forces’ ASDS is a battery-powered, dry interior  submersible that is carried to a deployment area by specially configured  688-class submarines. ASDS is intended to provide increased range,  payload, on-station loiter time, endurance, and communication/sensor  capacity over current submersibles. The 65-foot-long, 8-foot-diameter  ASDS is operated by a two-person crew and includes a lock out/lock in  diving chamber. SOCOM is the resource sponsor and provides the  requirements and funding, and the Naval Sea Systems Command—the  Navy’s technical expert for major undersea systems—is the program  manager responsible for overseeing the prime contractor, Northrop  Grumman Corporation. Over the years, the ASDS acquisition milestone  decision authority has resided at various levels within DOD.
In 1994, the Navy awarded a $70 million cost-plus incentive fee contract to  Westinghouse Electric Corporation’s Oceanic Division in Annapolis,  Maryland, for detailed design, construction, testing, documentation,  refurbishment, and delivery of the first ASDS with the option to build one  or two more systems. In 1996, Northrop Grumman bought this division and  assumed responsibility for the Annapolis division’s performance on the  ASDS contract. In December 2005, ASDS program management lead was  reassigned to Northrop Grumman in Newport News, Virginia, which has  greater technical experience in submarines, and Northrop Grumman  Electronic Systems in Annapolis is assisting.
The original program’s schedule called for delivery of the first boat in July  1997. However, numerous technical problems with key subsystems  contributed to performance shortfalls, schedule delays, and cost increases.  In August 2001, the Navy program office took what it called “conditional”  preliminary acceptance of the first boat from Northrop Grumman under an  agreement that all requirements needed for final acceptance would be  completed within 1 year, requirements that the contractor was unable to  accomplish. On June 26, 2003, the Navy elected to accept the ASDS boat in  an “as is” condition, and incorporated additional waivers, deviations, and  engineering change proposals into the contract. As a result, acceptance of  the ASDS boat did not require any additional actions on the part of the  contractor. Further, the Navy did not seek any consideration from the  contractor because Navy officials believed at the time that the ASDS met  virtually all of its requirements. By that time, the total costs for the ASDS  development contract had already increased from $70 million to more than  $340 million.
In October 2003, following the Navy’s acceptance of ASDS, the Navy  negotiated and signed a basic ordering agreement (BOA) with Northrop  Grumman to provide a range of goods and services to support the ASDS  program. For example, the BOA enabled the Navy to order engineering  and design services; overhaul, repair, and inspection services; logistical  support; and spare parts and materials for a 3-year period. The BOA was  extended an additional year in 2006. To expedite the contracting process,  the BOA established specific labor rates for different types of service, such  as program office, technical, engineering, operations, and quality support.  Through March 2007, the Navy issued 26 delivery orders with an estimated  value of over $84 million. The duration of the current BOA extends  through September 2007, and the Navy anticipates awarding a new BOA  for another 2 years while overall ASDS performance is reevaluated. Under  another BOA, Northrop Grumman is also providing ASDS engineering  services, such as engineering changes and drawing updates, for  Portsmouth Naval Shipyard.
In assessing the ASDS program we drew heavily from our previous work  on best practices in defense acquisitions. This work has shown that both a  sound business case and effective contracting strategy are essential for  success. A sound business case involves firm requirements and mature  technologies, a knowledge-based acquisition strategy, realistic cost and  schedule estimates, and sufficient funding. An effective contracting  strategy involves selecting a contractor with proper expertise, choosing  contracting approaches that effectively balance risk, and effectively  managing and assessing contractor performance; all of which are intended  to promote accountability for outcomes and protect the taxpayers’  interests.
Critical flaws in the Navy’s initial business case contributed to ASDS’s  acquisition challenges and increased the government’s risk. We have  previously reported that the capabilities required of the boat outstripped  the contractor’s resources in terms of technical knowledge, time, and  money. The Navy’s overly optimistic assumptions about the contractor’s  ability to readily incorporate existing submersible and commercial  technology into the ASDS resulted in a mismatch between technologies  and needed capabilities and an ill-advised decision to combine  developmental and operational testing. Further information on the  technical, cost, and management issues that undermined the ASDS’s initial  business case may be found in appendix I.
section_title: Navy Assumed Responsibility for ASDS Problems through Its Decisions and Contracting Approach
paragraphs: As existing problems mounted during development and new ones arose  after acceptance, the Navy increasingly assumed responsibility for  resolving them. This responsibility required additional time and money  over the targets that had been established by the ASDS development  contract. Since accepting the ASDS in June 2003, SOCOM has continued to  invest millions of dollars to fix both old and new problems. The prime  contractor has had little incentive to control costs given the Navy’s choice  of certain cost-reimbursable contract types. Navy officials say they accept  more risk of performance because ASDS relies on new, highly technical  subsystems that are inherently risky. The Navy’s risk also increased  because it authorized work before reaching agreement on key contract  terms and conditions and failed to finalize them in a timely manner,  indicating a lack of discipline in the contracting process.
subsections: section_title: Over Time, the Navy Assumed the Cost and Responsibility for Correcting ASDS Problems
paragraphs: Resolving the flawed initial business case required additional time and  money, far exceeding the target cost and delivery time frames established  under the ASDS September 1994 development contract. For example, the  development contract was awarded for about $70 million with an expected  delivery date of the first ASDS boat in July 1997. When the contractor  proved unable to meet these time frames, the Navy found itself having to  rebaseline the program in 1998 and 1999, more than doubling the  estimated development cost and extending the delivery schedule by more  than 2 years. Ultimately the development cost almost quintupled.
During the course of ASDS’s development, the Navy gradually assumed  responsibility for addressing ASDS’s technical problems by awarding  separate contracts to other organizations to develop key components. The  contractor’s lack of expertise in key technologies, such as the propeller  and battery, contributed to the Navy’s decision to seek outside expertise to  develop alternative solutions. More information on these actions is  provided in appendix I.
The Navy finally accepted the first ASDS boat in June 2003 in an “as is”  condition. Since the June 2003 acceptance, however, SOCOM has  continued to invest millions of dollars to address old and new technical  and reliability issues. Through March 2007, the Navy has issued delivery  orders with an estimated value of about $84 million under the BOA with  Northrop Grumman. Much of the funding has been for efforts to correct  design deficiencies and to improve ASDS’s reliability.
section_title: Navy’s Cost-Reimbursable Contracts Provided Little Incentive to the Contractor to Control Costs
paragraphs: Arrangements that appropriately share risk, incentivize performance, and  provide for accountability promote successful acquisition outcomes. The  government can choose from a range of contract types available to it that  gives it flexibility to acquire goods and services. The selection of contract  type is generally a matter of risk allocation: fixed-price contracts place the  risks associated with performing the contract on the contractor; cost-type  contracts share the risk between the contractor and the government. The  risk associated with performance shifts between the parties depending on  the type of cost contract selected. In selecting the contract type, the  government must consider the difficulty of providing the goods and  services in the time allocated for contract performance. For example,  when the risks are minimal or can be predicted with an acceptable degree  of certainty, such as when the government and the contractor have  sufficient knowledge of the effort required, then the government uses a  fixed-price contract, and the contractor has full responsibility for the  performance costs and the resulting profit or loss. In contrast, when the  extent of product knowledge is more limited, the government uses a cost- reimbursable contract; the government assumes more risk and may try to  motivate the contractor’s performance by using various incentive or award  fee provisions.
Our review found that nearly all of the $84 million in design, integration,  and reliability improvement work authorized under the Navy’s October  2003 BOA with Northrop Grumman used some form of a cost-reimbursable  contract. About 6 percent were conducted under a fixed-price type  arrangement. Of the first 18 delivery orders issued through early May 2005,  14 were either cost-plus fixed fee or labor-hour orders. Cost-plus fixed fee  arrangements negotiate the fee at the inception of the contract and do not  vary with the actual costs incurred by the contractor. Labor-hour contracts  provide for direct labor hours at specified fixed rates that include wages,  overhead, general, and administrative expenses. As profit and other  expenses are already included in the rates charged to the government, the  orders provided no profit incentive for the contractor to control costs or  work efficiently. Correspondingly, our analysis found that the ASDS  contractor often exceeded the initial estimates of the time and cost  required to complete the work: 12 of the 26 delivery orders issued under  the BOA exceeded the initial cost estimates, while the delivery schedule  was extended on 20 of the 26 orders. Figure 1 shows the value of all  delivery orders and subsequent modifications by contract type through  March 2007, based on the year the order was initially issued.
Navy officials told us that they chose cost-plus fixed fee or labor-hour  orders, in part, because ASDS relied on many new and highly technical  subsystems that were inherently risky. The ASDS contracting officer told  us that the choice of cost-plus fixed fee or labor-hour orders reflected the  perceived risk in the efforts, that is, the technical requirements and the  work that needed to be done were not always well-defined or known in  advance. Navy officials reported, however, that to get the contractor to  more actively manage and be accountable for success, the Navy has  increased the use of award and incentive fee provisions on its cost-type  orders, placing at least some of the contractor’s potential fee at risk. For  example, Navy officials noted that two of the three delivery orders issued  in 2006—representing about 80 percent of the value of ASDS work ordered  under new delivery orders during the year—contained award or incentive  fee provisions. While the Navy officials acknowledged that it was too early  to quantify the results of these approaches, preliminary indications are  that the contractor’s performance has improved and that the arrangements  are providing sufficient risk sharing and monetary incentives to motivate  contractor performance. Further, the contracting officer anticipated that  the Navy would use more fixed-price arrangements as more experience is  developed with ASDS repair and maintenance requirements.
section_title: Authorizing Work before Reaching Agreement on Key Terms and Conditions Increased the Navy’s Risk
paragraphs: Our analysis also found that the Navy often initiated work using  undefinitized contract actions; that is, before the Navy and contractor had  reached agreement on key terms and conditions of the delivery order,  such as the scope of the work to be performed and the price of that work.  While this approach allows agencies to begin needed work quickly, it also  exposes the government to potentially significant additional costs and  risk. For example, in September 2006 we reported on how DOD addressed  issues raised by the Defense Contract Audit Agency in audits of Iraq- related contract costs. We found that DOD contracting officials were less  likely to remove costs questioned by auditors if the contractor had already  incurred those costs while the contract action was undefinitized.
Our analysis found that 10 of the 26 ASDS delivery orders—accounting for  about 14 percent of the work—were initiated as undefinitized contract  actions. In most cases, the Navy justified the use of this approach by  stating that the work needed to begin immediately to meet urgent  operational requirements. For 7 of these 10 orders, the Navy failed to  definitize the orders within the 180-day time frame required under defense  acquisition regulations, taking instead from 228 to 509 days. In three cases,  the Navy definitized the orders after the work had been completed.
The delivery order to replace the ASDS’s hydraulic reservoir illustrates the  need to clearly define the scope of the work, provide effective  management and oversight, and hold the contractor accountable for  outcomes. The delivery order issued to the contractor on June 10, 2005,  was a $1.0 million cost-plus fixed fee undefinitized contract to replace the  ASDS’s hydraulic reservoir. In October 2005, the contractor reported it  would need about $444,000 extra to complete the project. Rather than  provide additional funds, the Navy elected to reduce the scope of the  work, and the order was definitized on March 1, 2006—nearly 9 months  after the work was initially authorized—at a cost of about $937,000. Two  days later, the contractor reported that the projected cost of the work had  almost doubled to more than $1.85 million. In a letter to the contractor, the  Commander, Naval Sea Systems Command, noted that at no time during  negotiations had the contractor identified the potential cost growth.  Nevertheless, as of December 20, 2006, a further modification to the  delivery order increased the estimated cost to $2.8 million and extended  the delivery date by 60 days.
Navy officials acknowledged that the use of undefinitized contract actions  and the failure to definitize them in a timely fashion indicated a lack of  discipline in the contracting process, but noted that officials had taken a  number of actions to address the issues, including taking more time to  define requirements and requiring the contractor to submit more realistic  cost and schedule estimates. Furthermore, the Navy has not issued an  undefinitized contract action since July 2005.
section_title: DOD Evaluating Future Options for Meeting Required Capabilities
paragraphs: Continuing reliability problems led to a DOD decision to cancel purchases  of additional ASDS boats, following on an earlier decision to decertify  ASDS for operational test readiness because of considerable performance  and reliability issues that required significant additional resources for new  development, investigations, rework, and design changes. Instead, DOD  directed the establishment of an ASDS improvement program and an  assessment of alternate material solutions to fulfill remaining operational  requirements. The results of both should allow DOD to make an informed  decision as to its future needs by mid-2008.
subsections: section_title: Additional Procurements Canceled Because of Continuing Reliability Problems
paragraphs: The Navy decertified ASDS from operational test readiness in October  2005, following a propulsion-related failure during an attempt at follow-on  operational test and evaluation. This failure, however, was among a series  of performance and reliability issues identified over the course of ASDS  development. These performance and reliability problems have required  significant additional resources to support new development,  investigations, re-work, and design changes. Some changes have not been  fully corrected or verified in operational testing. For example, in  December 2003, while transporting ASDS mated to the host submarine,  severe damage occurred to the ASDS tail section—the propeller assembly,  the stator, and the stern planes. The Navy’s investigation attributed the  cause to improper maintenance procedures—inadequate assembly by  Portsmouth Naval Shipyard personnel. The propeller assembly and stern  plane designs were improved and maintenance procedures were changed.  In June 2004 testing of repairs, however, the ASDS propeller stator broke  off and damaged the propeller. The investigation found that the stator had  been improperly manufactured by a subcontractor. The tail damage was  repaired by Northrop Grumman at the Navy’s expense. During follow-on  test and evaluation in October 2005, ASDS experienced a propulsion  system failure that was attributed to improper assembly/installation of the  new titanium tail.
Because of the investigations of the December 2003 and June 2004 ASDS  tail casualties, the Navy re-evaluated the effects of unsteady hydrodynamic  loads on the boat. Although neither casualty was attributed to this type of  load, the Navy determined that, due to fatigue stresses, the aluminum tail  was not structurally adequate to last the life of the ASDS. The tail was  replaced with a titanium and composite-based tail, but the replacement  has not resolved all the tail assembly design deficiencies. To minimize the  potential for damage to the tail, the Navy has imposed operating  restrictions that limit the speed of the host submarine while transporting  ASDS, which will remain in effect until this issue has been resolved.
In September 2005, the Navy and SOCOM chartered the ASDS Reliability  Action Panel (ARAP)—consisting of technical experts from government  and industry—to conduct an independent assessment of reliability. After  the 2005 propulsion system failure, the ARAP was asked to assess ASDS’s  readiness to resume testing. ARAP’s report indicated that there were  numerous examples of unpredicted component reliability problems and  failures resulting from design issues, and recommended not resuming  testing until detailed reviews of mission critical systems were completed.  In November 2005, SOCOM restructured the ASDS program to focus on  improving reliability of the existing boat before investing in additional  boats. The existing boat is currently available only for limited operational  use.
In April 2006, DOD canceled plans to procure follow-on ASDS boats and  directed the Navy and SOCOM to (1) establish an ASDS-1 improvement  program to increase the performance of the existing boat to the required  level, to insert technologies to avoid obsolescence, and to complete  operational testing and (2) assess alternate material solutions to fulfill  remaining operational requirements. In May 2006, DOD reported to the  congressional defense committees that the first ASDS would be  maintained as an operational asset, and that an ASDS improvement  program was planned through fiscal year 2008. As currently structured, the  ASDS reliability improvement program includes four elements    ASDS Phase 1 and Phase 2 critical systems reviews,  reliability builds or upgrades, and    verification testing.
The results of the Phase 1 critical systems review are due in June 2007 and  are expected to include prioritized corrective actions and associated cost  and schedule estimates. According to Navy officials, the Phase 1 results  are expected to identify critical upgrades to improve reliability and make  ASDS-1 a viable operational asset.
At-sea tests to verify that corrections result in improved performance and  reliability are being conducted. In October 2006 ASDS completed a  successful 2-week underway period operating from a host submarine to  verify and test repairs that were made to the propulsion system. In  February and March 2007, following installation of 15 reliability  improvements, including a newly designed hydraulic reservoir and  environmental control unit, ASDS verification testing was conducted.   This testing consisted of nine underways for a total 113 operating hours.  According to SOCOM, there were no failures. Follow-on operational test  and evaluation is scheduled for the second half of fiscal year 2008. It is not  certain, however, the extent to which the upgrades identified by the Phase  1 critical systems review will be incorporated into the ASDS for this  operational test.
section_title: DOD Expects to Make a Program Decision in mid-2008
paragraphs: DOD also directed the Navy and SOCOM to conduct an assessment of  alternate material solutions to fulfill remaining operational requirements.  An independent cost and capability trade study is under way for the  purpose of developing models for both the ASDS and a hybrid combatant  submersible to support concept design-level trade studies. A final report is  expected by the end of June 2007. SOCOM has completed a requirements  analysis that identified undersea clandestine maritime mobility gaps for  special operations forces insertion and extraction as well as the conduct  of undersea tasks. According to SOCOM, in February 2007, it submitted a  memorandum on these issues to DOD’s Joint Staff for submission to the  Joint Requirements Oversight Council (JROC). Upon JROC approval, the  memorandum is expected to serve in-lieu of an Initial Capabilities  Document for use in the alternate material solutions analysis. This process  is similar to an analysis of alternatives and is expected to assess a broad  range of potential material solutions. The joint Navy-SOCOM alternate  material solutions analysis is expected to be completed by February 2008.
A program decision is planned in mid-2008, after the ASDS improvement  program and alternate material solutions analysis are completed.
According to SOCOM and Navy officials, the results of the alternate  material solutions analysis, in conjunction with the operational testing of  the changes made in response to the reliability improvement program,  should provide DOD by mid-2008 with sufficient information to make an  informed decision on the direction DOD should take to meet its  operational needs.
section_title: Conclusions
paragraphs: Had the original business case for ASDS been properly assessed as an  under-resourced, concurrent technology, design, and construction effort  led by an inexperienced contractor, DOD may have adopted an alternative  solution or strategy. Ironically, after having invested about $885 million in  nearly 13 years, DOD may still face this choice. As to lessons learned,  DOD’s actions to make the boat operational came at great expense to the  government. Further, DOD’s inadequate program and contract  management in essence made the prime contractor’s poor performance  acceptable. These actions underscore the need to have a sound business  case at the start of a program, coupled with an acquisition strategy that  enables the government to alter course as early as possible. Instilling more  discipline into the contracting process is a step in the right direction, but  its success hinges on DOD’s willingness to hold the contractor  accountable. From this point forward, DOD will be conducting reviews  and testing to guide its decisions on how to proceed with the first ASDS  boat. It is important that DOD be guided by sound criteria and a sound  contracting strategy as it makes these decisions.
section_title: Recommendations
paragraphs: We are making three recommendations. In order to prevent the  government from accepting additional undue risks and expense on ASDS,  the Secretary of Defense should:    Establish acceptable cost, schedule, and performance criteria, based  on fully defined scopes of work, and assess the boat’s ability to meet  these criteria at the Phase 1 and Phase 2 critical systems reviews and at  the management reviews. If, by the time of the program decision in  mid-2008, ASDS does not meet acceptable cost, schedule, or  performance criteria, we recommend that the Secretary of Defense  discontinue the effort and not proceed with further tests.
Ensure that, if the review results meet acceptable cost, schedule, and  performance criteria, the design changes resulting from the Phase 1  critical systems review essential for demonstrating ASDS reliability and  maintainability be incorporated in sufficient time to be tested under  operational conditions prior to the planned mid-2008 decision on how  to best meet special operations forces’ requirements.
Require the Navy to include provisions in the ASDS contracting  strategy chosen when the existing BOA expires that (1) appropriately  balance risk between the government and the contractor through the  contract types selected, (2) incentivize the contractor’s performance  and promote accountability for achieving desired outcomes by properly  structuring the award and incentive fees, and (3) provide the kind of  management and oversight of the program necessary to hold the  contractor accountable for performance.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD partially concurred with our first two recommendations that it  establish acceptable cost, schedule, and performance criteria for ASDS-1;  assess the boat’s ability to meet these criteria; and test design changes.  DOD concurred with our third recommendation on the Navy’s contracting  strategy to balance risk between the government and contractor; properly  structure award and incentive fees to incentivize contractor performance  and promote accountability; and provide necessary management and  oversight to hold the contractor accountable. DOD’s written comments are  reprinted in appendix II.
In partially concurring with our first recommendation, DOD commented  that under its new ASDS management plan, program decisions will be  made through management reviews using specified evaluation criteria and  not solely at the completion of the critical systems reviews. The Navy  provided a copy of its March 6, 2007 management plan for ASDS-1  improvement. This plan represents a positive step in establishing a  structured strategy for the ASDS-1 improvement program, including  defining management oversight—roles, responsibilities, and authorities— and providing specific criteria to guide the program’s continuation or  termination decisions. However, the criteria may not go far enough.  Specifically, the criteria may not be sufficient for making an informed  program decision—the scope of the proposed ASDS’s critical systems  upgrades may not be fully defined and realistic cost and schedule  estimates may not be developed before the ASDS improvement effort is  approved to proceed. Further, the management plan does not address the  Phase 2 critical systems review decision. We have clarified this  recommendation to incorporate the management program reviews and  decisions and added language to focus more directly on the need for fully  defined scopes of work. Fully defining the scopes of work is key to  realistic cost and schedule estimates.
DOD partially concurred with our second recommendation, but took issue  with operationally testing all Phase 1 critical systems review design  changes before the planned mid-2008 decision. DOD stated that there are  identified changes that will take more time and that a decision on what  changes to implement will depend on various factors such as time,  funding, and scope. However, it remains unclear the extent to which  upgrades that affect performance will be incorporated and tested prior to  the mid-2008 program decision. We modified the wording to require testing  essential design changes prior to a 2008 decision.
DOD also provided technical comments, which we have incorporated as  appropriate.
section_title: Scope and Methodology
paragraphs: To assess the ASDS contracting strategy, we reviewed the ASDS  acquisition strategy, program documents, contract documentation, and  numerous historical documents, including the Navy’s 1997 Independent  Review Team assessment, the joint Navy/SOCOM 1999 Independent  Review Team assessment, and the ASDS Reliability Action Panel’s 2006  report. In our assessment of ASDS, we drew upon our large body of  previous work on best practices for developing products and developing  sound business cases. To determine the status of major ASDS technical  issues and program restructuring, we examined program status documents  and briefings, test results, technical reports, and various memos and  guidance. We did not assess the appropriateness of accepting the first  ASDS boat in an “as is” condition. In performing our work, we obtained  information and interviewed officials from the U.S. Special Operations  Command; the Naval Sea Systems Command’s ASDS program and  contracting offices; and the Navy’s Operational Test and Evaluation Force.  We conducted our review from July 2006 to April 2007 in accordance with  generally accepted government auditing standards.
We are sending copies of this report to the Secretary of Defense; the  Secretary of the Navy; the Commander, U.S. Special Operations Command;  the Director of the Office of Management and Budget; and interested  congressional committees. We will make copies available to others upon  request. In addition, the report will be made available at no charge on  GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact   me at (202) 512-4841 or by email at francisp@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found   on the last page of this report. Contributors to this report include  Catherine Baltzell, David Best, Timothy DiNapoli, David Hand,   John Krump, Mary Quinlan, and Robert Swierczek.
section_title: Appendix I: Mismatches in Technology, Resources, and Managerial Capacity Undermined Key Business Case Assumptions
paragraphs: Putting a development program on sound footing from the beginning  requires that the selected technology be capable of meeting the  government’s requirements and able to be developed within needed time  frames and available resources. Further, the contractor must have the  technical and managerial capacity to effectively execute the contract,  while the government must be able to provide effective program and  management oversight. On the ASDS program, however, these conditions  were not present at the start of or effectively applied during the  development effort, undermining the ability to successfully design and  deliver an operational ASDS boat.
subsections: section_title: Technology Assumptions Optimistic
paragraphs: A key to promoting successful acquisition outcomes is matching available  resources with the requirements for the proposed system. Specifically, the  government must match its needs with technology that has been proven to  work in a realistic environment before committing to production. In this  case, the Navy assumed that the conceptual design was technically sound  and that the design would incorporate a large amount of fully developed  submersible or commercially available technology. The Navy’s September  1993 acquisition strategy concluded that the low risk of integrating  technologies already in use on existing submarines and submersible  vehicles eliminated the need for an advanced development model or a  demonstration/validation phase with developmental and operational  testing. Further, the Navy determined that by concurrently addressing  manufacturing and test issues during the design process, lengthy redesign  periods would be avoided. Consequently, in September 1994, the Assistant  Secretary of the Navy for Research, Development, and Acquisition (the  designated program decision authority) approved Milestone II  (development) and replaced a sequential test program (development tests,  operational tests, technical evaluations, and operational test and  evaluation) with a consolidated and integrated test program. At the same  time, the ASDS program’s Milestone III (production decision) was waived  because of the limited number of procurement quantities.
The Navy’s confidence in the maturity of technology also played a large  role in its assessment of proposed designs for the ASDS, and in turn, in its  selection of the contractor. The Navy concluded that the contractor’s  conceptual design exceeded various requirements, and, based on its  maturity, the proposed design approach was low risk. From the outset,  the Navy’s assessments of the contractor’s design solution, experience,  and management capabilities proved incorrect. Incorporating commercial  off the shelf components into the ASDS was more challenging than  expected. For example, the contractor had difficulty understanding  underwater shock performance requirements and eventually  subcontracted the shock design efforts to a specialty firm.
During the course of ASDS’s development, the Navy gradually assumed  responsibility for addressing ASDS’s technical problems by awarding  separate contracts to other organizations to develop key components. The  contractor’s lack of expertise in key technologies, such as the propeller  and battery, contributed to the Navy’s decision to seek outside expertise to  develop alternative solutions. In turn, the Navy provided these  components to Northrop Grumman as government-furnished equipment,  accepting both the cost and the risk for their performance and paid  Northrop Grumman millions of dollars to integrate the components onto  the ASDS boat. These actions include the following examples:    The ASDS program has invested over $26 million since 2000 to design,  develop, and integrate a new lithium-ion battery to replace the  inadequate silver-zinc battery provided by the prime contractor. In  October 2000, the Navy awarded Northrop Grumman a $2.1 million  contract modification to design, develop, test, and incorporate a  lithium-ion polymer battery. By September 2003, a series of contract  modifications had increased the cost of the prototype battery effort to  $5.9 million and had extended delivery until February 28, 2004. The  Navy sought other experts to identify and test an alternative lithium-ion  battery that could be housed in the existing ASDS titanium battery  bottles. In May 2004, after evaluating three proposals, the Navy  awarded Yardney Technical Products a $9.3 million contract for a  complete ASDS shipset battery that was delivered in 2005. To date, the  Navy has provided Northrop Grumman more than $6 million to  integrate the lithium ion battery.
The Navy invested over $1.5 million to redesign the first ASDS  propeller, which was a major source of noise during testing. Rather  than task Northrop Grumman to redesign the propeller, the Navy  awarded a $1.5 million contract in 2002 to Pennsylvania State  University’s Applied Research Laboratory to design and build a new  composite propeller. Northrop Grumman installed this propeller in  April 2003 at a cost of about $140,000. Pennsylvania State University  has since provided two additional propellers at a cost of about  $576,000.
section_title: Initial Cost Projections Harbingers of Difficulties to Come
paragraphs: Another key to successful acquisition outcomes is to accurately estimate  the resources needed to develop and produce a system. The Navy had  information before awarding the ASDS contract indicating, however, that  the contractor’s proposed price might not be realistic. Specifically, the  contract’s negotiated price was about 60 percent less than the Navy’s  November 1993 cost and operational effectiveness analyses. The Navy’s  price evaluation team concluded that the contractor’s proposed amounts  for ASDS development and production were underestimated and that  overruns were likely. Among the lessons learned cited by two independent  review teams in 1997 and 1999 were that the program was underfunded, in  part because the Navy did not give sufficient weight to concerns raised by  cost analysts, and that the contractor “bid to the budget.”
section_title: Government and Contractor Management Was Ineffective
paragraphs: The government’s and contractor’s capacity to effectively manage a  program is another key determinant in promoting successful outcomes.  The Navy concluded in 1994 that overall, the contractor’s design,  management capabilities, and cost control capabilities were equal to or  better than the two other competitors for the ASDS program and that the  contractor had adequate experience in submersible design, construction,  and certification. This assessment, as well as the government’s capacity to  provide effective management and oversight of the ASDS program, soon  proved incorrect.
The Navy’s 1997 and the joint Navy/SOCOM 1999 independent review  teams identified weaknesses in the contractor’s capacity to effectively  address technical issues and manage the ASDS program. One team noted  that the contractor had considerable difficulty in interpreting the  underwater shock portion of the ASDS performance requirements. The  teams attributed these difficulties, in part, to the contractor’s lack of  experience in submarine design, in contrast to the initial business case  assumption. Further, the reviews noted that the Navy’s review of the  contractor’s design products revealed that substandard design  methodology was used, resulting in unacceptable system design. The  review teams also found that this lack of experience had a detrimental  effect on the contractor’s overall ability to understand technical nuances  and may have prevented the contractor from applying appropriate  management attention when needed. For example, the contractor used  two different systems for reporting and managing the program; the  contractor’s cost reports contained errors; and its estimates to complete  the effort were updated only every 6 months, resulting in unanticipated  and sudden cost increases being reported to the Navy. Additionally, the  contractor constrained its estimates by imposing “management  challenges,” which the team concluded were in reality artificial reductions  imposed by the contractor to obscure the contractor’s problems and  mislead attempts to analyze its projected costs.
Further, the review teams concluded that lapses in effective management  by both the government and the contractor contributed to the program’s  challenges. The teams identified several causes for these lapses, including    a lack of contractor experience in submarine design and construction;  the government’s lack of influence or visibility into problems between  the contractor and the subcontractors;    a focus on technical rather than management aspects of the program  by both the program office and the contractor;  ineffective oversight by the program office and little attention to the  financial performance of the contractor; and  frequent changes in the contractor’s project management team.
The Navy program office and Northrop Grumman have taken steps to  improve the program’s management. In 2005, the Naval Sea Systems  Command reorganized the program office for a greater emphasis on  special operations programs. In December 2005, Northrop Grumman  reassigned the ASDS program’s management lead to its Newport News  division, which has greater management and technical experience in  submarines. Northrop Grumman Newport News is leading the Phase 1  ASDS critical systems review, and Northrop Grumman Electronic Systems  is assisting.
section_title: Appendix II: Comments from the Department of Defense