Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy
Published: 2006-04-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) is planning to invest $1.3 trillion between 2005 and 2009 in researching, developing, and procuring major weapon systems. How DOD manages this investment has been a matter of congressional concern for years. Numerous programs have been marked by cost overruns, schedule delays, and reduced performance. Over the past 3 decades, DOD's acquisition environment has undergone many changes aimed at curbing cost, schedule, and other problems. In order to determine if the policy DOD put in place is achieving its intended goals, we assessed the outcomes of major weapons development programs initiated under the revised policy. Additionally, we assessed whether the policy's knowledge-based, evolutionary principles are being effectively implemented, and whether effective controls and specific criteria are in place and being used to make sound investment decisions.
section_title: What GAO Found
paragraphs: Changes made in DOD's acquisition policy over the past 5 years have not eliminated cost and schedule problems for major weapons development programs. Of the 23 major programs we assessed, 10 are already expecting development cost overruns greater than 30 percent or have delayed the delivery of initial operational capability to the warfighter by at least 1 year. The overall impact of these costly conditions is a reduction in the value of DOD's defense dollars and a lower return on investment. Poor execution of the revised acquisition policy is a major cause of DOD's continued problems. DOD frequently bypasses key steps of the knowledge-based process outlined in the policy, falls short of attaining key knowledge, and continues to pursue revolutionary--rather than evolutionary or incremental--advances in capability. Nearly 80 percent of the programs GAO reviewed did not fully follow the knowledge-based process to develop a sound business case before committing to system development. Most of the programs we reviewed started system development with immature technologies, and half of the programs that have held design reviews did so before achieving a high level of design maturity. These practices increase the likelihood that problems will be discovered late in development when they are more costly to address. Furthermore, DOD's continued pursuit of revolutionary leaps in capability also runs counter to the policy's guidance. DOD has not closed all of the gaps in the policy that GAO identified nearly 3 years ago, particularly with regard to adding controls and criteria. Effective controls require decision makers to measure progress against specific criteria and ensure that managers capture key knowledge before moving to the next acquisition phase. However, DOD's policy continues to allow managers to approach major investment decisions with many unknowns. Without effective controls that require program officials to satisfy specific criteria, it is difficult to hold decision makers or program managers accountable to cost and schedule targets. In this environment, decision-making transparency is crucial, but DOD is lacking in this area as well.

section_title: Letter
paragraphs: DOD’s planned investment in research, development, and procurement of  major weapon systems will total approximately $1.3 trillion between 2005  and 2009, with over $800 billion of that investment yet to be made. DOD is  facing a significant number of problems in managing its acquisitions.  Military operations in Afghanistan and Iraq are consuming a large share of  DOD resources and causing the department to invest more money sooner  than expected to replace or fix existing weapons. Meanwhile, DOD is  intent on transforming military operations while pursuing multiple  megasystems that are expected to be the most expensive and complex  ever. These costly conditions coupled with increases in spending for other  national priorities, such as health care and social security, make it  essential that DOD effectively leverage its investments, particularly in  weapon system acquisitions. If DOD manages its current portfolio of  weapons within traditional margins of error, the financial consequences  could be dire.
DOD’s strategy for acquiring major weapon systems has traditionally been  to plan programs that would achieve a big leap forward in capability  within a single development program, a strategy that often results in major  cost and schedule problems. We have assessed weapon acquisitions as a  high-risk area for 15 years, and although U.S. weapons are among the best  in the world, the programs to acquire them have continued to produce  poor cost and schedule outcomes. However, the current defense  acquisition environment continues to be characterized by cost and  schedule growth, a lack of confidence by congressional and DOD leaders,  and no appreciable improvement in the defense acquisition system. DOD  knows what to do to achieve better outcomes. It has written into policy an  approach that advocates that adequate knowledge be attained at critical  junctures before DOD managers agree to invest more money in the next  phase of weapon system development. The policy also emphasizes  evolutionary principles for acquiring weapons rather than trying to achieve  a big leap forward in capability within a single development program. We  have reported in the past that DOD’s revised policy does not incorporate  adequate controls to ensure the effective implementation of a knowledge- based, evolutionary acquisition process. However, DOD believes that the  policy includes the necessary controls to achieve effective outcomes.
You requested that we evaluate DOD’s compliance with and  implementation of its revised acquisition policy intended to produce better  cost, schedule, and performance outcomes for major acquisition  programs. In order to obtain an early assessment of the cost and schedule  impact of the revised policy, and to assess DOD’s effectiveness in  implementing a knowledge-based, evolutionary acquisition approach we  assessed (1) the cost and schedule status of major weapons development  programs initiated under the revised policy, (2) whether the policy’s  knowledge-based, evolutionary acquisition principles are being effectively  implemented, and (3) whether effective controls and specific criteria are  in place and being used to make sound investment decisions.
In conducting our evaluation, we reviewed pertinent acquisition statutes,  policies, and guidance; analyzed development cost and schedule data for  23 major acquisition programs approved to start system development  under DOD’s revised acquisition policy between October 2000 and  December 2004; conducted case study reviews of nine of those   23 programs; and interviewed officials from the Office of Secretary of  Defense and each of the military services. We conducted our review from  May 2005 to February 2006 in accordance with generally accepted  government auditing standards. Additional information about our  methodology is contained in appendix I.
subsections: section_title: Background
paragraphs: Historically, DOD’s programs for acquiring major weapon systems have  taken longer, cost more, and often delivered fewer quantities and other  capabilities than planned. GAO has documented these problems for  decades. In 1970, GAO reported that considerable cost growth had been  and was continuing to occur on many current development programs.  Since that report was issued, numerous changes have been made to DOD’s  acquisition process and environment to try to improve acquisition  outcomes. Those changes include numerous executive branch initiatives  and legislative actions as well as roughly 11 revisions to DOD’s acquisition  policy between 1971 and 2005. Despite these efforts, defense acquisition  programs in the past 3 decades continued to routinely experience cost  overruns, schedule slips, and performance shortfalls.
Figure 1 illustrates the continued problem of development cost overruns.  The figure depicts the combined cost overruns for large development  programs (programs totaling more than $1 billion for research,  development, testing and evaluation in fiscal year 2005 dollars) in each of  the past 3 decades. The figure also identifies some of the major studies and  improvement efforts initiated during this time frame. As the figure  illustrates, efforts to improve acquisition outcomes have not been  successful in curbing acquisition cost problems. Programs initiated in the  1970s exceeded DOD’s initial investment estimate by 30 percent, or   $13 billion (in fiscal year 2005 dollars), and similar outcomes continued  during the subsequent decades despite numerous reform efforts and policy  revisions.
Since the mid-1990s, we have studied the best practices of leading  commercial companies. Taking into account the differences between  commercial product development and weapons acquisitions, we  articulated a best practices product development model that relies on  increasing knowledge when developing new products, separating  technology development from product development, and following an  evolutionary or incremental product development approach. This  knowledge-based approach requires developers to make investment  decisions on the basis of specific, measurable levels of knowledge at  critical junctures before investing more money and before advancing to  the next phase of acquisition. An evolutionary product development  process defines the individual increments on the basis of mature  technologies and a feasible design that are matched with firm  requirements. Each increment should be managed as a separate and  distinct acquisition effort with its own cost, schedule and performance  baseline. An increment that excludes one of these key elements puts an  extra burden on decision makers and provides a weak foundation for  making development cost and schedule estimates. The knowledge-based,  evolutionary approach in our model is intended to help reduce  development risks and to achieve better program outcomes on a more  consistent basis.
Hoping to improve acquisition outcomes, DOD leaders initiated significant  revisions to the department’s acquisition policy again in October 2000, by  adopting the knowledge-based, evolutionary system development  approach. We reported in November 2003, that much of the revised policy  agrees with GAO’s extensive body of work and that of successful  commercial firms. DOD’s revised policy emphasizes the importance of and  provides a good framework for capturing knowledge about critical  technologies, product design, and manufacturing processes. If properly  implemented and enforced this approach could help DOD’s decision  makers gain the confidence they need to make significant and sound  investment decisions for major weapon systems. Furthermore, the policy’s  emphasis on evolutionary system development sets up a more manageable  environment for achieving knowledge. We also noted that DOD’s policy  strongly suggests the separation of technology development from system  development, a best practice that helps reduce technological risk at the  start of a program and makes cost and delivery estimates much more  predictable. Figure 2 depicts in general how DOD’s revised policy adopts  key aspects of the best practices model.
Although DOD took significant steps in the right direction, its policy does  not include controls that require program officials to meet the key criteria  that we believe are necessary for ensuring that acceptable levels of  knowledge are actually captured before making additional significant  investments. We previously recommended that DOD design and  implement necessary controls to ensure that appropriate knowledge is  captured and used to make decisions about moving a program forward  and investing more money at critical junctures. DOD officials  acknowledged the advantages of using knowledge-based controls, but  stated that they believed the policy already included enough controls to  achieve effective program results. The officials agreed to monitor the  acquisition process to assess the effectiveness of those controls and to  determine whether additional ones are necessary.
section_title: DOD’s Revised Policy Has Not Improved Development Program Outcomes
paragraphs: The cost and schedule outcomes being achieved by development programs  initiated since DOD first issued its revised policy have not improved over  those achieved by programs managed under prior versions of the policy.  Of the 23 major programs we assessed, 10 have already reported estimated  development cost growth greater than 30 percent or expected delays of at  least 1 year in delivery of an initial operational capability to the warfighter.  These programs combined represent a cost increase of $23 billion (in  fiscal year 2005 dollars) and an average delay in delivery of initial  capability of around 2 years. Most of the other programs were still in the  early stages as of December 2005 with over half of system development  remaining and had not yet reported an adequate amount of cost or  schedule data to effectively analyze their progress. Table 1 contains the  cost and schedule increases for the 23 programs we assessed, expressed  as a percentage of each program’s development estimate.
The Army’s Future Combat System is a case in point. Less than 3 years  after program initiation and with $4.6 billion invested, the Army has  already increased its development cost estimate $8.9 billion or 48 percent  and delayed delivery of initial capability by 4 years over the original  business case. Similarly, just over 1 year after initiating development of the  Aerial Common Sensor aircraft, the Army has reported that severe weight  and design problems discovered during development have stopped work  on the program. As a result, program officials are anticipating at least a   45 percent cost increase and a delay of 2 years in delivering an initial  capability to the warfighter. These two Army programs are not the only  ones experiencing problems. Table 2 contains cost and schedule data for   6 of the 10 largest development programs initiated under the revised  policy, including the Future Combat System and Aerial Common Sensor.  As the table illustrates there are several programs experiencing large cost  increases and schedule delays.
A good measure of acquisition performance is return on investment as  expressed in acquisition program unit cost because unit cost represents  the value DOD is getting for its acquisition dollars invested in a certain  program. The programs listed in table 2 will not achieve the return on  investment that DOD anticipated when they began development. In the  case of Joint Strike Fighter, for example, DOD initially intended to  purchase 2,866 aircraft at an acquisition program unit cost of about   $66 million. The Navy has reduced the number of Joint Strike Fighter  aircraft it plans to buy; technology and design problems encountered  during development have led to the significant cost growth. As a result, the  acquisition program unit cost is now about $84 million, an increase of   27 percent. We recently reported that the risk of even greater increases is  likely because flight testing has not yet started and the acquisition strategy  involves substantial overlap of development and production. Similar  problems have led to increases in the Future Combat System program. At  program initiation, the Army anticipated that each of 15 units would cost  about $5.5 billion to develop and deliver. Since that time, instability in the  program’s technologies and requirements have led to significant cost  increases, leading to a 54 percent increase in acquisition program unit  cost, now estimated to be $8.5 billion.
Regarding all 23 development programs, DOD leaders originally planned to  invest a total of about $83 billion (fiscal year 2005 dollars) for system  development and anticipated delivering an initial operational capability to  the warfighter in 77 months on average. However, development costs have  grown and delivery schedules have been delayed significantly. DOD now  expects to invest over $106 billion in those same programs, an increase of  over $23 billion or 28 percent. The delivery of initial capability to the  warfighter is expected to take an average of 88 months or nearly 1 year  longer than originally planned. Figure 3 shows changes in these business  case elements for these programs in the short time since their initiation.
section_title: DOD Is Not Effectively Implementing the Policy’s Knowledge- Based, Evolutionary Approach
paragraphs: DOD is not effectively implementing the knowledge-based process and  evolutionary approach emphasized in its acquisition policy. While the  policy outlines a specific knowledge-based process of concept refinement  and technology development to help ensure a sound business case is  developed before committing to a new development program, almost   80 percent of the programs we reviewed were permitted to bypass this  process. Furthermore, the policy emphasizes the need to mature all critical  technologies before starting system development and to demonstrate that  the product’s design is mature before beginning system demonstration.  However, nearly three-fourths of the programs reported having immature  critical technologies when they received approval to start development,  and at least half of the programs had not achieved design maturity before  holding their design review and gaining approval to enter the system  demonstration phase of development. The policy also emphasizes the use  of an evolutionary product development approach, yet program officials  continue to structure major acquisition programs to achieve large  advances in capability within a single step development program. This  strategy has historically resulted in poor cost and schedule outcomes.
subsections: section_title: Knowledge-Based Process Not Enforced
paragraphs: DOD decision makers continue to approve programs for system  development that have not followed key elements of the policy’s suggested  knowledge-based process. The policy requires program managers to  provide senior decision makers with knowledge about key aspects of a  system at critical investment points in the acquisition process. Our prior  reviews have identified those critical points as the start of system  development or program start (referred to as Milestone B in the DOD  acquisition policy), design readiness review separating system integration  and system demonstration, and production commitment (Milestone C in  the DOD acquisition policy). The most important point occurs at program  start, when system development begins. DOD acquisition guidance  emphasizes the importance of the acquisition phases preceding program  start, noting that the decisions made during those phases—concept  refinement and technology development—generally define the nature of  an entire acquisition program.
subsections: section_title: Acquisition Officials Are Not Effectively Using Early Processes to Develop Executable Business Cases
paragraphs: Acquisition officials continue to begin system development without  following early processes for developing executable business cases. A  business case should provide demonstrated evidence that (1) the  warfighter’s needs are real and necessary and that they can best be met  with the chosen concept and (2) the chosen concept can be developed and  produced within existing resources—including technologies, design  knowledge, funding, and the time to deliver the product when it is needed.  Establishing a business case calls for a realistic assessment of risks and  costs; doing otherwise undermines the intent of the business case and  invites failure. This process requires the user and developer to negotiate  whatever trade-offs are needed to achieve a match between the user’s  requirements and the developer’s resources before system development  begins.
The revised policy and associated guidance emphasize the importance of  following a sound process of systems engineering and decision making  prior to initiating a system development program. The process established  in the policy consists of two phases, concept refinement and technology  development, and a major decision review called Milestone A, which if  rigorously followed, would provide acquisition officials with an  opportunity to assess whether program officials had the knowledge  needed to develop an executable business case. However, almost   80 percent of the programs we reviewed began system development  without holding any prior decision review. Senior officials with the Office  of the Secretary of Defense confirmed that this is a common practice  among defense acquisition programs. This practice eliminates a key  opportunity for decision makers to assess early product knowledge  needed to establish a business case that is based on realistic cost,  schedule, and performance expectations.
Although program officials conduct analysis before starting a development  program, they do not consistently follow a process to capture the critical  knowledge needed to produce executable business cases, as evidenced by  the poor outcomes current programs are experiencing. Officials with the  Office of the Secretary of Defense recognized this lack of rigor and  discipline in acquisition process, and in February 2004, the Under  Secretary of Defense (Acquisition, Technology and Logistics) issued a  department-wide policy memorandum directing acquisition officials to  place greater emphasis on systems engineering when planning and  managing acquisition programs. The policy requires programs to develop a  systems engineering plan that describes the programs’ overall technical  approach, including processes, resources, metrics, and applicable  performance incentives. Although DOD’s systems engineering initiative  has the potential to improve program performance, officials have found  that the preliminary results are mixed. Early analysis shows that  implementation is inconsistent while program officials learn to develop  and implement systems-engineering plans.
section_title: Programs Continue to Enter System Development with Immature Technologies
paragraphs: DOD decision makers continue to permit programs to enter system  development before critical technologies are mature. Our review of  technology readiness assessments and acquisition decision memorandums  for our nine case study programs found that seven of the nine programs  were approved to begin development even though program officials  reported levels of knowledge below the criteria suggested in the policy  and associated guidance, specifically in the area of technology maturity.  Those seven programs are not isolated cases. As illustrated in Figure 4,   13 of the programs (nearly three-fourths) that received approval to enter  system development under the revised policy did so with less than   100 percent of their critical technologies mature to the level specified by  DOD. Only 2 of those programs had more than 75 percent of their  technologies mature when they began (see appendix III for technology  maturity data for each program).
Even though acquisition policy states that technologies shall be mature  before beginning system development, the practice of accepting high  levels of technology risk at program start continues to be the norm and not  the exception. An official with Office of the Secretary of Defense  responsible for reviewing and validating program assessments of  technology maturity informed us that the office generally views immature  critical technologies at the beginning of development as an acceptable risk  as long as program officials can show that they have a plan to mature the  technologies by the time the program reaches its design readiness review,  which requires additional investments to move a program from system  integration into system demonstration. Therefore, risk management plans  are consistently viewed as acceptable substitutes for demonstrated  knowledge.
section_title: Programs Continue Past Design Reviews before Design Maturity is Demonstrated
paragraphs: In addition to emphasizing the importance of capturing technology  knowledge before starting system development, DOD’s policy also  highlights the importance of demonstrating design maturity before moving  from the integration phase of system development into system  demonstration and initial manufacturing. The policy establishes a design  readiness review between the two phases to determine whether a  product’s design is mature and stable and whether the product is ready to  move ahead. While DOD’s policy does not require programs to  demonstrate any specific level of design maturity, our past work has found  that a key indicator of design maturity is the completion of 90 percent of  the system’s engineering drawings. We found that defense programs that  moved forward with lower levels of design maturity, as indicated by  drawing completion, encountered costly design changes and parts  shortages that, in turn, caused labor inefficiencies, schedule delays, and  quality problems. Consequently, those programs required significant  increases in resources—time and money—over what was estimated at the  point each program entered the system demonstration phase.
We analyzed engineering drawing completion data for 8 programs initiated  under the revised policy that have held a design review, and found that  more than half of those programs had not completed 90 percent of their  design drawings before they received approval to enter the system  demonstration phase of development. We also analyzed drawing-release  data for three programs that have not yet held their design review but have  projected the number of drawings officials anticipate will be completed  when their reviews are held. Based on projections provided by program  officials, 2 of those 3 programs are expected to have less than 55 percent  of their drawings complete before they seek approval to begin system  demonstration and initial manufacturing.
section_title: Evolutionary Acquisition Is Not Being Used
paragraphs: Despite the revised policy’s guidance that capabilities should be developed  and delivered in individually defined and separately managed increments,  a majority of major weapon acquisition programs we assessed continue to  be structured to achieve revolutionary increases in capability within one  development program. According to the policy, the objective of an  evolutionary approach is to balance needs and available capability with  resources and put capability into the hands of the user quickly. The policy  states that the success of the strategy depends on consistent and  continuous definition of requirements and the maturation of technologies  that lead to disciplined development and production of systems that  provide increasing capability. In this approach, requirements that cannot  be satisfied within these limits as well as available financial resources  must wait for future generations of the product and be managed as  separate system development programs with separate milestones, costs,  and schedules. In our case studies of nine acquisition programs initiated  under the revised policy, we found only one program—the Small Diameter  Bomb—that satisfied all of the criteria of an evolutionary approach. In five  case studies, we found that program officials had claimed that their  programs were evolutionary, yet our evidence shows they were not  evolutionary in practice; and in three cases, program officials chose not to  use evolutionary acquisition from the outset. Table 3 summarizes our  assessment of the nine case studies.
section_title: Specific Criteria Are Needed to Ensure Disciplined and Transparent Investment Decisions
paragraphs: The revised acquisition policy does not contain effective controls that  require the demonstration of product knowledge measured against  specific criteria to ensure that acquisition officials make disciplined,  transparent, and knowledge-based investment decisions. The lack of  specific required criteria creates an environment in which unknowns  about technology, design, and manufacturing processes are acceptable.  Decision makers and program officials are left with no objective measures  against which to gauge a program’s level of knowledge, making  accountability difficult. In the absence of criteria, transparency in  acquisition decisions is essential to ensuring accountability, but key  decision documents do not provide sufficient information about major  decisions. DOD believes that acquisition decision memorandums, used to  document program decisions, provide adequate transparency. However,  the decision memorandums we reviewed did not contain an explanation of  the decision maker’s rationale and rarely identify remaining risks,  especially as they relate to the key knowledge standards emphasized in the  policy. Further, the timeliness, accessibility, and depth, of the data  contained in the Selected Acquisition Reports, DOD’s primary means of  providing Congress with a status report of program performance, inhibits  the reports’ usefulness as a management and oversight tool.
In November 2003, we reported that the revised acquisition policy lacked  many of the controls that leading commercial companies rely on to attain  an acceptable level of knowledge before making additional significant  investments. Controls are considered effective if they are backed by  specific criteria and if decision makers are required to consider the  resulting data before deciding to advance a program to the next level.  Controls used by leading companies help decision makers gauge progress  in meeting cost, schedule, and performance goals and hold program  managers accountable for capturing relevant product knowledge to inform  key investment decisions. The controls we have articulated as best  practices used by successful commercial product developers are listed  below in table 5.
Some senior officials with the Office of the Secretary of Defense believe  that the effective use of controls in DOD’s policy and the establishment of  more specific criteria for decision making would improve program  outcomes. They note that specific criteria need to be established and that  programs need to be held accountable to those criteria before being  permitted to proceed into the next phase. They also note that the criteria  for moving an acquisition effort from one phase of the process to the next,  primarily documented in acquisition decision memorandums as exit  criteria, are not typically specific and often do not relate to the key  knowledge-based criteria suggested in the policy.
We found this to be true for our nine case study programs. We reviewed  acquisition decision memorandums in our case studies and determined  that they were not useful in explaining the decision maker’s rationale and  in almost all of the cases they did not address the key knowledge criteria  suggested in the acquisition policy. In most instances, the decision maker  simply noted that the program being assessed was ready to proceed into  system development, but did not provide an explanation of the rationale  for the decision. Senior officials with the Office of the Secretary of  Defense told us that they agree that a better explanation of the decision  maker’s rationale, specifically in instances where the knowledge criteria  are not fully met, would provide transparency and ultimately allow for a  more accountable decision-making process. The following two examples  illustrate how decision documentation is lacking:    The Future Combat System program received approval to enter system  development and demonstration in 2003, with 19 percent of its critical  technologies mature, well below the policy’s standard. The acquisition  decision memorandum supporting this decision did not provide the  rationale for approving the system with such a large number of immature  critical technologies. The memo did direct an updated review of the  decision 18 months later and that the program “remain flexible and open  to accommodate trades in the system architecture and in the individual  systems’ designs.”
The Joint Strike Fighter program was approved to enter system  development in 2001. The acquisition decision memorandum did not  address the fact that 75 percent of the program’s critical technologies were  not mature to the policy’s standard. The memorandum did acknowledge  that the program’s requirements could be changed or modified, noting that  further refinements in the requirements should be explored as a potential  way to reduce program costs. However, the memorandum did not explain  why the decision maker determined that the program should enter  development without achieving the technology and requirements  knowledge emphasized in the policy.
The acquisition decision memorandums for most of the other programs we  reviewed did not specifically address critical gaps in knowledge, nor did  they effectively explain the decision makers’ rationale for deeming those  programs ready to begin system development. In memos where we found a  reference to key knowledge principles, such as technology maturity, the  decision makers acknowledged that more effort was needed to meet the  policy’s suggested criteria but considered the risk acceptable to begin  development. These memos did not explain why risks were considered  acceptable. For example, the Navy’s Multi-Mission Maritime Aircraft  program had none of its critical technologies mature at program initiation.  The decision maker acknowledged the need to further mature the critical  technologies but approved the program to enter development. Instead of  holding the program to the policy’s criteria for entering development, the  decision maker simply directed the Navy to work with the Office of the  Secretary of Defense to implement risk mitigation and technology  maturation plans during the integration phase of system development.
In addition to the lack of transparency provided through acquisition  decision memoranda, we also found that the data presented to Congress in  DOD’s Selected Acquisition Reports (SARs) provided only limited  usefulness as an oversight tool. Since 1969, SARs have been the primary  means by which DOD reports the status of major weapon system  acquisitions to Congress. SARs are reports that are expected to contain  information on the cost, schedule, and performance of major weapon  systems in comparison with baseline values established at program start,  full-scale development, and production decision points. Our analysis, as  well as a previous GAO review, of current and historical SAR data found  that the timeliness, accessibility, and depth of the data contained in the  reports limits their usefulness as an oversight tool. Our prior review noted  that a number of opportunities exist for DOD to give Congress more  complete information on the performance of major defense acquisition  programs. DOD agreed that SAR data could be improved to make it more  useful to Congress.
section_title: Conclusions
paragraphs: Failing to consistently implement the knowledge-based process and  evolutionary principles emphasized in the revised acquisition policy— coupled with a lack of specific criteria for making key investment  decisions—are keeping DOD on its historical path of poor cost and  schedule outcomes. Most programs are incurring the same scope of cost  overruns and schedule delays as programs managed under prior DOD  policies. More consistent use of the early acquisition processes would  improve the quality and viability of program business cases by ensuring  they are founded on knowledge obtained from rigorous and disciplined  analysis. The initiative by Office of the Secretary of Defense to reinstitute  the use of systems engineering is a step in the right direction. However, in  order for this initiative to be effective DOD must establish and enforce  specific criteria at key decision points. Our past work has identified and  recommended criteria and controls that should be consistently applied at  major decision points. The enforcement of these criteria is critical to  ensuring that programs have the knowledge necessary to successfully  move forward through the acquisition process. DOD officials have  acknowledged the advantages of using knowledge-based criteria and  controls, but believe the policy already includes enough controls to  achieve effective program results. However, without enforceable criteria,  defense officials are challenged to determine whether adequate knowledge  has been obtained for investing taxpayer dollars. The lack of enforceable  criteria also makes it difficult to hold defense officials accountable for  their decisions.
section_title: Recommendations for Executive Action
paragraphs: DOD must ensure that appropriate knowledge is captured and used at  critical junctures to make decisions about moving a program forward and  investing more money. We recommend that the Secretary of Defense  require program officials to demonstrate that they have captured  appropriate knowledge at three key points—program start, design review  for transitioning from system integration to system demonstration, and  production commitment—as a condition for investing resources. At a  minimum those controls should require program officials to demonstrate  that they have achieved a level of knowledge that meets or exceeds the  following criteria at each respective decision point:    Program start (Milestone B): Start of product development    Demonstrate technologies to high readiness levels    Ensure that requirements for the product are informed by the systems-   Establish cost and schedule estimates for product on the basis of  knowledge from preliminary design using system engineering tools    Conduct decision review for program start   Design readiness review: Beginning of system demonstration    Complete 90 percent of design drawings    Complete subsystem and system design reviews    Demonstrate with prototype that design meets requirements    Obtain stakeholders’ concurrence that drawings are complete and    Complete the failure modes and effects analysis    Identify key system characteristics   Identify critical manufacturing processes     Establish reliability targets and growth plan on the basis of  demonstrated reliability rates of components and subsystems    Conduct decision review to enter system demonstration   Production commitment (Milestone C): Initiation of low-rate  production    Demonstrate manufacturing processes    Build production-representative prototypes    Test production-representative prototypes to achieve reliability goal    Test production-representative prototypes to demonstrate product in    Collect statistical process control data    Demonstrate that critical processes are capable and in statistical    Conduct decision review to begin production  Furthermore, to ensure that major decisions are transparent and that program  officials and decision makers are held accountable, we recommend that the  Secretary of Defense require decision makers to include written rationale for  each major decision in acquisition decision documentation. The rationale  should address the key knowledge-based criteria appropriate for milestone  decisions, explain why a program’s level of knowledge in each area was  deemed acceptable if criteria have not been met and provide a plan for  achieving the knowledge necessary to meet criteria within a given time frame.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided written comments on a draft of this report. The comments  appear in appendix II.
DOD partially concurred with our recommendation that the Secretary of  Defense should establish specific controls to insure that program officials  demonstrate that they have captured a level of knowledge that meets or  exceeds specific criteria at three key points in the acquisition process:  program start, design readiness review, and production commitment. DOD  agreed that knowledge-based decision making is consistent with sound  business practice and stated that it would continue to develop policy that  reflects a knowledge-based approach and improves acquisition outcomes.  DOD noted that it would consider our recommendations as it reassesses  the DOD acquisition business model and the knowledge required at each  decision point. We believe that DOD’s plan to reassess its business model  provides a good opportunity to establish the controls and specific criteria  recommended in this report. Therefore, we are retaining our  recommendation that the Secretary of Defense should establish controls  to insure that program officials demonstrate that they have captured a  level of knowledge that meets or exceeds specific criteria at three key  points in the acquisition process.
DOD also partially concurred with our recommendation that the Secretary  of Defense require decision makers to provide written rationale in  acquisition decision documentation for each major decision. DOD agreed  that acquisition decisions should be documented, decision makers should  be held accountable, and that they should provide the rationale for their  decisions. DOD believes that the implementation of Section 801 of the  National Defense Authorization Act for FY 2006 reinforces these  processes. The act calls for the decision maker to certify that the program  meets certain requirements, such as technology maturity, prior to starting  a new development program at Milestone B. However, the act is focused  on the decision to start a development program and does not identify  specific criteria for programs to be measured against at design readiness  review or production commitment. We believe our recommendation adds  transparency and accountability to the process because it requires the  decision maker to provide the rationale for a decision to allow a program  to move forward, not only at Milestone B but at other key decision points  as well. Therefore, we are retaining our recommendation that the  Secretary of Defense require decision makers to provide written rationale  for each major decision in acquisition decision documentation.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Air Force, Army, and Navy; and the Director of the  Office of Management and Budget. We will provide copies to others on  request. This report will also be available at no charge on GAO’s Web site  at http://www.gao.gov.
If you have any questions about this report or need additional information,  please call me at (202) 512-4841 (sullivanm@gao.gov). Contact points for  the offices of Congressional Relations and Public Affairs are located on  the last page of this report. Key contributors to this report were Michael  Hazard, Assistant Director; Lily Chin; Ryan Consaul; Christopher DePerro;  Travis Masters; and Adam Vodraska.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess the impact of DOD’s revised acquisition policy, we analyzed cost  and schedule data for 23 major defense acquisition programs that were  approved to begin system development under the revised policy. We did  not assess space, missile defense, or ship programs. We collected our data  from Selected Acquisition Reports, presidential budget documents,  ongoing GAO work, and pertinent program officials. We utilized previous  GAO reports related to defense acquisition policies and worked with  knowledgeable GAO staff to ensure the use of current, accurate data. We  also analyzed more than 150 annual Selected Acquisition Reports covering  a 36-year period from 1969 to 2005, to determine historical trends related  to outcomes of acquisition policy implementation.
We assessed whether the revised policy’s knowledge-based, evolutionary  acquisition principles were being effectively implemented by conducting   9 case study reviews and analyzing design maturity data for 11 programs  that have made engineering-drawing data available to GAO. Our case study  programs were the Aerial Common Sensor, Multi-Platform Radar  Technology Insertion Program, Global Hawk Unmanned Aerial Vehicle,  Small Diameter Bomb, Future Combat System, Joint Strike Fighter,  Expeditionary Fighting Vehicle, Multi-Mission Maritime Aircraft, and the  E-2 Advanced Hawkeye. We interacted directly with numerous program  officials to seek input on current developments with their programs. We  studied program documents to assess how well programs understand and  are implementing the revised acquisition policy. We also analyzed drawing  release data for those programs that have either passed their design  review or have provided GAO with estimated drawing release data for a  future design review to assess design maturity. In several cases, we asked  that program offices verify information in these various documents.
We also reviewed Department of Defense (DOD) Directive 5000.1, DOD  Instruction 5000.2, and the Defense Acquisition Guidebook. In addition we  examined each of the military services’ policy directives and guidance,  DOD memorandums to include policy intent and DOD expectations  regarding policy implementation as well as Joint Capabilities Integration  and Development System documents. We interviewed relevant officials in  Washington, D.C., from the Office of the Director, Defense Research and  Engineering, the Joint Staff, the Office of the Secretary of Defense, and  Army, Navy, and Air Force acquisition policy staff in order to better  understand the content of these documents and the intent of DOD’s policy.
We conducted our review from May 2005 to February 2006 in accordance  with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: Program Data for 23 Programs Initiated under DOD’s Revised Acquisition Policy (as of December 2005)
paragraphs: Appendix III: Program Data for 23 Programs  Initiated under DOD’s Revised Acquisition  Policy (as of December 2005)
subsections: section_title: Program start
paragraphs: Formal  Milestone I or  Milestone A  decision  review?
Program office projections.
section_title: Related GAO Products
paragraphs: DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.  Washington, D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February  11, 2003  Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,  2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,  2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,  2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April  26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,  D.C.: August16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,  D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon  System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,  1996.