Defense Acquisitions: Observations on Weapon Program Performance and Acquisition Reforms
Published: 2010-05-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The past two years have seen the Congress and DOD take meaningful steps towards addressing long-standing weapon acquisition issues--an area that has been on GAO's high risk list since 1990. This testimony focuses on the progress DOD has made in improving the planning and execution of its weapon acquisition programs and the potential for recent acquisition reforms to improve program outcomes. The testimony includes observations about (1) DOD's efforts to manage its portfolio of major defense acquisition programs, (2) the knowledge attained at key junctures of a subset of 42 weapon programs from the 2009 portfolio, (3) other factors that can affect program execution, and (4) DOD's implementation of recent acquisition reforms. The testimony is based on the results of our annual assessment of weapon programs. To conduct the assessment, GAO analyzed data on the composition of DOD's portfolio of major defense acquisition programs. GAO also collected data from program offices on technology, design, and manufacturing knowledge, as well as on other factors that can affect program execution. GAO has made numerous recommendations on weapon system acquisition in prior work but is not making any new recommendations in this testimony.
section_title: What GAO Found
paragraphs: While DOD still faces significant challenges in managing its weapon system programs, the current acquisition reform environment provides an opportunity to leverage the lessons of the past and manage risks differently. This environment is shaped by significant acquisition reform legislation, constructive changes in DOD's acquisition policy, and initiatives by the administration, including making difficult decisions to terminate or trim numerous weapon systems. To sustain momentum and make the most of this opportunity, it will be essential that decisions to approve and fund acquisitions be consistent with the reforms and policies aimed at getting better outcomes. DOD has started to reprioritize and rebalance its weapon system investments. In 2009, the Secretary of Defense proposed canceling or significantly curtailing weapon programs with a projected cost of at least $126 billion that he characterized as too costly or no longer relevant for current operations, while increasing funding for others that he assessed as higher priorities. Congress supported several of the recommended terminations. DOD plans to replace several of the canceled programs in fiscal years 2010 and 2011, hopefully with new, knowledge-based acquisition strategies, because the warfighter need remains. The most significant of these will be the effort to restructure the Army's terminated Future Combat System program. At the same time, however, DOD's portfolio of major defense acquisition programs continues to grow. Between December 2007 and July 2009, the number of major defense acquisition programs grew from 96 to 102 programs. GAO has previously reported that DOD should continue to work to balance its weapon system portfolio with available funding, which includes reducing the number or size of weapon system programs, or both, and assessing the affordability of new programs and capabilities in the context of overall defense spending. At the program level, our recent observations present a mixed picture of DOD's adherence to a knowledge-based acquisition approach, which is a key for improving acquisition outcomes. For 42 programs GAO assessed in depth in 2010, there has been continued improvement in the technology, design, and manufacturing knowledge programs had at key points in the acquisition process. However, most programs are still proceeding with less knowledge than best practices suggest, putting them at higher risk for cost growth and schedule delays. A majority of programs have also experienced requirements changes, software development challenges, or workforce issues, or a combination, which can affect program stability and execution. DOD has begun to implement a revised acquisition policy and congressional reforms that address many of these areas. For example, eight programs we examined in the technology development phase plan to test competitive prototypes before starting system development and seven programs plan to hold early systems engineering reviews. If DOD consistently applies this policy, the number of programs adhering to a knowledge-based acquisition should increase and the outcomes for DOD programs should improve.

section_title: Letter
paragraphs: I am pleased to be here today to discuss the Department of Defense’s  (DOD) management of its acquisition of major weapons systems—an area  that has been a part of GAO’s high-risk list since 1990—and the potential  for recent acquisition reforms to improve program outcomes. While DOD  still faces significant challenges in managing its weapon system programs,  the past two years have seen DOD and the Congress take meaningful steps  towards addressing long-standing weapon acquisition issues. DOD made  major revisions to its acquisition policies to place more emphasis on  acquiring knowledge about requirements, technology, and design before  programs start—thus putting them in a better position to field capabilities  on-time and at the estimated cost. Congress strengthened DOD’s  acquisition policies and processes by passing the Weapon Systems  Acquisition Reform Act of 2009, which includes provisions to ensure  programs are based on realistic cost estimates and to terminate programs  that experience high levels of cost growth. The House Armed Services  Committee Panel on Defense Acquisition Reform issued its final report  and made recommendations on areas, such as assessing the performance  of the defense acquisition system, that were incorporated into the  proposed Implementing Management for Performance and Related  Reforms to Obtain Value in Every Acquisition (IMPROVE) Act of 2010. In  addition, DOD has started to reprioritize and rebalance its weapon system  investments. In DOD’s fiscal year 2010 and 2011 budget requests, the  Secretary of Defense proposed ending all or part of at least a half dozen  major defense acquisition programs that were over cost, behind schedule,  or no longer suited to meet the warfighters’ current needs. Congress’s  support for several of the recommended terminations signaled a  willingness to make difficult choices on individual weapon systems and  DOD’s weapon system investments as a whole.
While DOD’s acquisition policies and process may be headed in the right  direction, fiscal pressures continue to build. Notwithstanding the federal  government’s long-term fiscal challenges, the Pentagon faces its own near- term and long-term fiscal pressures as it attempts to balance competing  demands, including ongoing operations in Afghanistan and Iraq, initiatives  to grow and modernize the force, and increasing personnel and health care  costs. While DOD’s fiscal year 2010 budget request started the process of  reprioritizing acquisition dollars to meet warfighters’ most pressing needs,  the department must still address the overall affordability of its weapon  system investments. DOD should continue to work to balance its weapon  system portfolio with available funding, which includes reducing the  number or size of weapon system programs, or both, and assessing the  affordability of new programs and capabilities in the context of overall  defense spending.
My statement focuses on the progress DOD has made in improving the  planning and execution of its weapon acquisition programs and the  potential for recent acquisition reforms to improve program outcomes. It  includes observations about (1) DOD’s efforts to manage its portfolio of  major defense acquisition programs, (2) the knowledge attained at key  junctures of a subset of 42 weapon programs from the 2009 portfolio,   (3) other factors that can affect program execution, and (4) DOD’s  implementation of recent acquisition reforms. The testimony is based on  the results of our recently issued annual assessment of weapon programs.  To conduct the assessment, GAO analyzed data on the composition of  DOD’s portfolio of major defense acquisition programs. GAO also  collected data from program offices on technology, design, and  manufacturing knowledge, as well as on other factors that can affect  program execution. That work was conducted in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Observations on DOD’s 2009 Major Defense Acquisition Program Portfolio
paragraphs: In our 2010 assessment of weapon programs, we made several  observations concerning DOD’s management of its major defense  acquisition portfolio. First, in DOD’s fiscal year 2010 budget, the Secretary  of Defense proposed canceling or significantly curtailing programs with  projected total costs of at least $126 billion that he characterized as too  costly or no longer relevant for current operations, while increasing  funding for others that he assessed as higher priorities. Congress  supported several of the recommended terminations (see table 1).
Second, DOD plans to replace several of the canceled programs in fiscal  years 2010 and 2011, hopefully with new, knowledge-based acquisition  strategies, because the warfighter need remains. The most significant of  these new programs will be the effort to restructure the Army’s Future  Combat System program into several smaller, integrated programs. Third,  DOD’s portfolio of major defense acquisition programs grew to 102  programs in 2009—a net increase of 6 since December 2007. Eighteen  programs with an estimated cost of over $72 billion entered the portfolio.  Not all of these programs entering the portfolio are new starts. For  instance, the Airborne Signals Intelligence Payload, and the Reaper  Unmanned Aircraft System are two programs that began as acquisition  category II programs, but their total research and development or  procurement costs now exceed the threshold for major defense  acquisition programs. Twelve programs with an estimated cost of $48  billion, including over $7 billion in cost growth, left the portfolio. These  programs left the portfolio for a variety of reasons, including program  restructure, termination, or completion. When the Future Combat System  is added to the programs leaving the portfolio, the total cost of these  programs increases to $179 billion, including over $47 billion in cost  growth.
Our 2010 assessment did not include an analysis of the cost and schedule  performance of DOD’s major defense acquisition program portfolio as a  whole. In recent years, this analysis showed that the cumulative cost  growth on DOD programs had reached $300 billion (in fiscal year 2010  dollars) and the average delay in delivering initial capabilities was 22  months. DOD did not issue timely or complete Selected Acquisition  Reports for its major defense acquisition programs in fiscal year 2009 for  the second consecutive presidential transition, which precluded an  analysis of the performance of DOD’s portfolio. We will resume our  portfolio analysis in next year’s assessment.
section_title: Observations from Our Assessment of Knowledge Attained by Key Junctures in the Acquisition Process
paragraphs: At the program level, our recent observations present a mixed picture of  DOD’s adherence to a knowledge-based acquisition approach, which is  key for improving acquisition outcomes. In our 2010 assessment of  weapon programs, we assessed the knowledge attained by key junctures  in the acquisition process for 42 individual weapon programs in DOD’s  2009 portfolio. While program knowledge is increasing, as in the past,  none of the 42 programs we assessed have attained or are on track to  attain all of the requisite amounts of technology, design, and production  knowledge by each of the key junctures in the acquisition process.  However, if DOD consistently implements its December 2008 policy  revisions on new and ongoing programs, then DOD’s performance in these  areas, as well as its cost and schedule outcomes, should improve. Our  analysis allows us to make five observations about DOD’s management of  technology, design, and manufacturing risks and its use of testing and  early systems engineering to reduce these risks.
Newer programs are beginning with higher levels of technology  maturity, but they are not taking other steps, such as holding  early systems engineering reviews, to ensure there is a match  between requirements and resources. Achieving a high level of  technology maturity by the start of system development is an important  indicator of whether a match between the warfighter’s requirements  and the available resources—knowledge, time, and money—has been  made. Since 2006, there has been a significant increase in the  percentage of technologies demonstrated in a relevant or realistic  environment by the start of system development. This increase  coincided with a change in statute. In 2006, the National Defense  Authorization Act included a provision requiring all major defense  acquisition programs seeking milestone B approval—entry into system  development—to get a certification stating the program’s technologies  have been demonstrated in a relevant environment. While only one of  the six programs that entered system development since 2006 and  provided data had fully mature critical technologies—that is,  demonstrated in a realistic environment, according to our criteria—all  the programs had critical technologies that had been at least  demonstrated in a relevant environment. Overall, only 4 of the 29  programs in our assessment that provided data on technical maturity at  development start did so with fully mature critical technologies.
While the technology levels of DOD programs entering system  development have increased, these programs are still not regularly  conducting early systems engineering reviews, which help ensure there  is a match between requirements and resources. We have previously  reported that before starting development, programs should hold  systems engineering events, such as the preliminary design review, to  ensure that requirements are defined and feasible and that the  proposed design can meet those requirements within cost, schedule,  and other system constraints. We have also found that programs  conducting these events prior to development start experienced less  research and development cost growth and shorter delays in the  delivery of initial operational capabilities than programs that  conducted these reviews after development start. Almost all nonship  programs (37 of 40 that provided data) in our latest assessment have  held at least one of three key systems engineering reviews (system  requirements review, system functional review, and preliminary design  review). However, only 1 of 37 programs that held a preliminary design  review did so before the start of system development. The remaining  programs held the review, on average, 30 months after development  start. The Weapon Systems Acquisition Reform Act of 2009 established  a statutory requirement for programs to conduct a preliminary design  review before milestone B, so we expect improvements in this area.
Programs that have held critical design reviews in recent years  reported higher levels of design knowledge; however, few  programs are demonstrating that the design is capable of  meeting performance requirements by testing an integrated  prototype. Knowing a product’s design is stable before system  demonstration reduces the risk of costly design changes occurring  during the manufacturing of production-representative prototypes— when investments in acquisitions become more significant. The overall  design knowledge that programs have demonstrated at their critical  design reviews has increased since 2003. Programs in our assessment  that held a critical design review between 2006 and 2009 had, on  average, almost 70 percent of their design drawings releasable at the  time of the review, which is a consistent upward trend since 2003.  However, most designs are still not stable at this point. Of the 28  programs in our latest assessment that have held a system-level critical  design review, only 8 reported having a stable design. Only 2 of the 5  programs that held a critical design review in 2009 had a stable design  at that point. The 5 programs reported that, on average, 83 percent of  the total expected drawings were releasable.
While the design knowledge of DOD programs at the system-level  critical design review has increased since 2003, these programs are still  not regularly demonstrating that these designs can meet performance  requirements by testing integrated prototypes before the critical design  review—a best practice. None of the 5 programs in our latest  assessment that held their critical design review in 2009 and planned to  test a prototype did so before the review. Of the 33 programs that  reported that they either had tested or were going to test an early  system prototype and provided a critical design review date, only 4 did  so before their critical design review. The Weapon Systems  Acquisition Reform Act of 2009 requires that DOD policy ensure that  the acquisition strategy for each major defense acquisition program  provides for competitive prototypes before milestone B approval,  unless a waiver is properly granted. This requirement should increase  the percentage of programs demonstrating that the system’s design  works as intended before the critical design review.
Some programs are taking steps to bring critical manufacturing  processes into control, however many programs still rely on “after  the fact” metrics. Capturing critical manufacturing knowledge before  entering production helps ensure that a weapon system will work as  intended and can be manufactured efficiently to meet cost, schedule,  and quality targets. Identifying key product characteristics and the  associated critical manufacturing processes is a key initial step to  ensuring production elements are stable and in control. Seven  programs in our latest assessment have identified their critical  manufacturing processes, including four of the programs that entered  production in 2009. Three of those seven programs reported that their  critical manufacturing processes were in control. It is generally less  costly—in terms of time and money—to eliminate product variation by  controlling manufacturing processes than to perform extensive  inspection after a product is built. However, many DOD programs rely  on inspecting produced components instead of using statistical process  control data in order to assess the maturity of their production  processes. For example, 12 programs in our assessment reported  tracking defects in delivered units, nonconformances, or scrap/rework  as a way to measure production process maturity. The use of “after the  fact” metrics is a reactive approach towards managing manufacturing  quality as opposed to a prevention-based approach.
Programs are still not regularly testing production  representative prototypes before committing to production. We  have previously reported that in addition to demonstrating that the  system can be built efficiently, production and postproduction costs  are minimized when a fully integrated, capable prototype is  demonstrated to show that the system will work as intended and in a  reliable manner. The benefits of testing are maximized when the tests  are completed prior to a production decision because making design  changes after production begins can be both costly and inefficient.  However, of the 32 programs in our assessment that could have tested  a prototype before production, only 17 either tested or expect to test a  fully configured, integrated, production-representative prototype  before holding their production decision. In December 2008, DOD  changed its policy to require programs to test production- representative articles before entering production.
More programs are using reliability growth curves before  beginning production. Reliability growth testing provides visibility  over how reliability is improving and uncovers design problems so  fixes can be incorporated before production begins. According to  DOD’s acquisition policy, a major defense acquisition program may not  proceed beyond low-rate initial production until it has demonstrated  acceptable reliability. Over half—22 of 40 programs that responded to  our questionnaire—reported that they use a reliability growth curve,  with 18 of these programs reporting they are currently meeting their  established goals. In addition, 12 of 19 programs that expect to hold  their production decision in 2010 and beyond reported using reliability  growth curves and most stated they are currently meeting their goals.  This practice should help these programs begin production with a  reliable product design.
section_title: Observations on Other Factors That Can Affect Program Execution
paragraphs: Our 2010 assessment of weapon programs also included three  observations on other areas related to DOD’s management of its weapons  programs, including requirements, software management, and program  office staffing. We have previously identified requirements changes and  increases in software lines of code as sources of program instability that  can contribute to cost growth and schedule delays. We have also reported  that workforce challenges can hinder program execution and negatively  affect program management and oversight.
A majority of programs changed key systems requirements after  development start. Of the 42 programs in our 2010 assessment that  reported tracking requirements changes, 23 programs reported having  had at least one change (addition, reduction, enhancement, or  deferment) to a key performance parameter—a top-level  requirement—since development start. Further, nine programs  experienced at least one change to a key system attribute—a lower  level, but still a crucial requirement of the system. Eight programs  reported major effects on the program as a result of these requirements  changes, such as not meeting acquisition program baseline cost,  schedule, and performance thresholds. DOD’s revised December 2008  acquisition policy attempts to reduce potentially disruptive  requirements changes by requiring programs to hold annual  configuration steering board meetings to ensure that significant  technical changes are not approved without considering their effect on  cost and schedule.
Many programs are at risk for cost growth and schedule delays  because of software development issues. Seventeen of the 28  programs in our 2010 assessment that reported data on software lines  of code estimated that the number of lines of code required for the  system to function has grown or will grow by 25 percent or more—a  predictor of future cost and schedule growth. Overall, the average  growth or expected growth in lines of code for the 28 programs was  about 92 percent. In addition to measuring growth in software lines of  code, we have previously reported that collecting earned value  management data for software development and tracking and  containing software defects in phase are good management practices.  Overall, 30 programs in our assessment reported collecting earned  value management data to help manage software development. Thirty- two programs in our latest assessment also reported collecting some  type of software defect data. For the 22 programs that responded a  more specific question about defect correction, on average, only 69  percent of the defects were corrected in the phase of software  development in which they occurred. Capturing software defects in  phase is important because discovering defects out of phase can cause  expensive rework later in programs.
Programs’ reliance on nongovernment personnel continues to  increase in order to make up for shortfalls in government  personnel and capabilities. In recent years, Congress and DOD have  taken steps to ensure the acquisition workforce has the capacity,  personnel, and skills needed to properly perform its mission; however,  programs continue to struggle to fill all staff positions authorized. Only  19 of the 50 programs in our 2010 assessment that responded to our  questions on staffing were able to fill all the positions they had been  authorized. A commonly cited reason for not being able to fill positions  was difficulty finding qualified candidates. As a result of staff shortfalls,  program offices reported that program management and oversight has  been degraded, contracting activities have been delayed, and program  management costs have increased as contractors are used to fill the  gap. Overall, 43 programs or 86 percent of those providing data  reported utilizing support contractors to make up for shortfalls in  government personnel and capabilities.
In addition, for the first time since we began reporting on program  office staffing in 2008, programs reported having more nongovernment  than government staff working in program offices (see table 2). The  greatest numbers of support contractors are in engineering and  technical positions, but their participation has increased in all areas,  from program management and contracting to administrative support  and other business functions.
section_title: Observations about DOD’s Implementation of Acquisition Reforms
paragraphs: DOD has begun to incorporate acquisition reforms into the acquisition  strategies for new programs. Both DOD’s December 2008 acquisition  policy revisions and the Weapon Systems Acquisition Reform Act of 2009  require programs to invest more time and resources in the front end of the  acquisition process—refining concepts through early systems engineering,  developing technologies, and building prototypes before starting system  development. In addition, DOD policy requires establishment of  configuration steering boards that meet annually to review all program  requirements changes as well as to make recommendations on proposed  descoping options that could help keep a program within its established  cost and schedule targets. These steps could provide a foundation for  establishing sound, knowledge-based business cases for individual  weapon programs and are consistent with many of our past  recommendations; however, if reform is to succeed and weapon program  outcomes are to improve, they must continue to be reinforced in practice  through decisions on individual programs.
Our analysis of the programs in our 2010 assessment allowed us to make  two observations about the extent to which DOD is implementing recent  acquisition reforms:    Most of the ten programs in our 2010 assessment that had not  yet entered system development reported having acquisitions  strategies consistent with both DOD’s revised acquisition policy  and the provisions of the Weapon Systems Acquisition Reform  Act of 2009. Specifically, 8 programs in our assessment planned to  develop competitive prototypes before milestone B. In addition, 7  programs have already scheduled a preliminary design review before  milestone B.
Only a few programs reported holding configuration steering  boards to review requirements changes, significant technical  changes, or de-scoping options in 2009. Seven programs in our  assessment reported holding configuration steering boards in 2009.  Under DOD’s revised acquisition policy, ongoing acquisition category I  and IA programs in development are required to conduct annual  configuration steering boards to review requirements changes and  significant technical configuration changes that have the potential to  result in cost and schedule effects on the program. In addition, the  program manager is expected to present de-scoping options to the  board that could reduce program costs or moderate requirements.  None of the programs reported that the boards that were held  approved requirements changes or significant technical changes. One  program—the P-8A Poseidon—reported that it presented de-scoping  options to decrease cost and schedule risk on the program and had  those options approved.
section_title: Concluding Observations on the Challenges to Achieving Lasting Reform
paragraphs: I would like to offer a few thoughts about other factors that should be  considered so that we make the most out of today’s opportunity for  meaningful change. First, I think it is useful to think of the processes that  affect weapon system outcomes (requirements, funding, and acquisition)  as being in a state of equilibrium. Poor outcomes—delays, cost growth,  and reduced quantities—have been persistent for decades. If we think of  these processes as merely “broken”, then some targeted repairs should fix  them. I think the challenge is greater than that. If we think of these  processes as being in equilibrium, where their inefficiencies are implicitly  accepted as the cost of doing business, then the challenge for getting  better outcomes is greater. Seen in this light, it will take considerable and  sustained effort to change the incentives and inertia that reinforce the  status quo.
Second, while actions taken and proposed by DOD and Congress are  constructive and will serve to improve acquisition outcomes, one has to  ask the question why extraordinary actions are needed to force practices  that should occur normally. The answer to this question will shed light on  the cultural or environmental forces that operate against sound  management practices. For reforms to work, they will have to address  these forces as well. For example, there have been a number of changes to  make cost estimates more rigorous and realistic, but do these address all  of the reasons why estimates are not already realistic? Clearly, more  independence, methodological rigor, and better information about risk  areas like technology will make estimates more realistic. On the other  hand, realism is compromised as the competition for funding encourages  programs to appear affordable. Also, when program sponsors present a  program as more than a weapon system, but rather as essential to new  fighting concepts, pressures exist to accept less than rigorous cost  estimates. Reform must recognize and counteract these pressures as well.
Third, decisions on individual systems must reinforce good practices.  Programs that have pursued risky and unexecutable acquisition strategies  have succeeded in winning approval and funding. If reform is to succeed,  then programs that present realistic strategies and resource estimates  must succeed in winning approval and funding. Those programs that  continue past practices of pushing unexecutable strategies must be denied  funding before they begin. This will require sustained leadership from the  Secretary of Defense, the Under Secretary of Defense for Acquisition,  Technology and Logistics, and the military services, and the cooperation  and support of Congress.
Fourth, consideration should be given to setting some limits on what is a  reasonable length of time for developing a system. For example, if a  program has to complete development within 5 or 6 years, this could serve  as a basis to constrain requirements and exotic programs. It would also  serve to get capability in the hands of the warfighter sooner.
Fifth, the institutional resources we have must match the outcomes we  desire. For example, if more work must be done to reduce technical risk  before development start—milestone B—DOD needs to have the  organizational, people, and financial resources to do so. Once a program is  approved for development, program offices and testing organizations must  have the workforce with the requisite skills to manage and oversee the  effort. Contracting instruments must be used that match the needs of the  acquisition and protect the government’s interests. Finally, DOD must be  judicious and consistent in how it relies on contractors.
Mr. Chairman, this completes my prepared statement. I would be happy to  respond to any questions you or other Members of the Subcommittee may  have at this time.
section_title: Contacts and Acknowledgements
paragraphs: For further information about this statement, please contact Michael J.  Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this statement. Individuals who made key contributions to this  statement include Ronald E. Schwenn, Assistant Director, Kristine R.  Hassinger, Carol T. Mebane, and Kenneth E. Patton.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.