Defense Acquisitions: Observations on Weapon Program Performance and Acquisition Reforms
Released: 2010-05-19T00:00:00

Letter

Observations on DOD’s 2009 Major Defense Acquisition Program Portfolio

In our 2010 assessment of weapon programs, we made several observations concerning DOD’s management of its major defense acquisition portfolio. First, in DOD’s fiscal year 2010 budget, the Secretary of Defense proposed canceling or significantly curtailing programs with projected total costs of at least $126 billion that he characterized as too costly or no longer relevant for current operations, while increasing funding for others that he assessed as higher priorities. Congress supported several of the recommended terminations (see table 1).

Second, DOD plans to replace several of the canceled programs in fiscal years 2010 and 2011, hopefully with new, knowledge-based acquisition strategies, because the warfighter need remains. The most significant of these new programs will be the effort to restructure the Army’s Future Combat System program into several smaller, integrated programs. Third, DOD’s portfolio of major defense acquisition programs grew to 102 programs in 2009—a net increase of 6 since December 2007. Eighteen programs with an estimated cost of over $72 billion entered the portfolio. Not all of these programs entering the portfolio are new starts. For instance, the Airborne Signals Intelligence Payload, and the Reaper Unmanned Aircraft System are two programs that began as acquisition category II programs, but their total research and development or procurement costs now exceed the threshold for major defense acquisition programs. Twelve programs with an estimated cost of $48 billion, including over $7 billion in cost growth, left the portfolio. These programs left the portfolio for a variety of reasons, including program restructure, termination, or completion. When the Future Combat System is added to the programs leaving the portfolio, the total cost of these programs increases to $179 billion, including over $47 billion in cost growth.

Our 2010 assessment did not include an analysis of the cost and schedule performance of DOD’s major defense acquisition program portfolio as a whole. In recent years, this analysis showed that the cumulative cost growth on DOD programs had reached $300 billion (in fiscal year 2010 dollars) and the average delay in delivering initial capabilities was 22 months. DOD did not issue timely or complete Selected Acquisition Reports for its major defense acquisition programs in fiscal year 2009 for the second consecutive presidential transition, which precluded an analysis of the performance of DOD’s portfolio. We will resume our portfolio analysis in next year’s assessment.

Observations from Our Assessment of Knowledge Attained by Key Junctures in the Acquisition Process

At the program level, our recent observations present a mixed picture of DOD’s adherence to a knowledge-based acquisition approach, which is key for improving acquisition outcomes. In our 2010 assessment of weapon programs, we assessed the knowledge attained by key junctures in the acquisition process for 42 individual weapon programs in DOD’s 2009 portfolio. While program knowledge is increasing, as in the past, none of the 42 programs we assessed have attained or are on track to attain all of the requisite amounts of technology, design, and production knowledge by each of the key junctures in the acquisition process. However, if DOD consistently implements its December 2008 policy revisions on new and ongoing programs, then DOD’s performance in these areas, as well as its cost and schedule outcomes, should improve. Our analysis allows us to make five observations about DOD’s management of technology, design, and manufacturing risks and its use of testing and early systems engineering to reduce these risks.

Newer programs are beginning with higher levels of technology maturity, but they are not taking other steps, such as holding early systems engineering reviews, to ensure there is a match between requirements and resources. Achieving a high level of technology maturity by the start of system development is an important indicator of whether a match between the warfighter’s requirements and the available resources—knowledge, time, and money—has been made. Since 2006, there has been a significant increase in the percentage of technologies demonstrated in a relevant or realistic environment by the start of system development. This increase coincided with a change in statute. In 2006, the National Defense Authorization Act included a provision requiring all major defense acquisition programs seeking milestone B approval—entry into system development—to get a certification stating the program’s technologies have been demonstrated in a relevant environment. While only one of the six programs that entered system development since 2006 and provided data had fully mature critical technologies—that is, demonstrated in a realistic environment, according to our criteria—all the programs had critical technologies that had been at least demonstrated in a relevant environment. Overall, only 4 of the 29 programs in our assessment that provided data on technical maturity at development start did so with fully mature critical technologies.

While the technology levels of DOD programs entering system development have increased, these programs are still not regularly conducting early systems engineering reviews, which help ensure there is a match between requirements and resources. We have previously reported that before starting development, programs should hold systems engineering events, such as the preliminary design review, to ensure that requirements are defined and feasible and that the proposed design can meet those requirements within cost, schedule, and other system constraints. We have also found that programs conducting these events prior to development start experienced less research and development cost growth and shorter delays in the delivery of initial operational capabilities than programs that conducted these reviews after development start. Almost all nonship programs (37 of 40 that provided data) in our latest assessment have held at least one of three key systems engineering reviews (system requirements review, system functional review, and preliminary design review). However, only 1 of 37 programs that held a preliminary design review did so before the start of system development. The remaining programs held the review, on average, 30 months after development start. The Weapon Systems Acquisition Reform Act of 2009 established a statutory requirement for programs to conduct a preliminary design review before milestone B, so we expect improvements in this area.

Programs that have held critical design reviews in recent years reported higher levels of design knowledge; however, few programs are demonstrating that the design is capable of meeting performance requirements by testing an integrated prototype. Knowing a product’s design is stable before system demonstration reduces the risk of costly design changes occurring during the manufacturing of production-representative prototypes— when investments in acquisitions become more significant. The overall design knowledge that programs have demonstrated at their critical design reviews has increased since 2003. Programs in our assessment that held a critical design review between 2006 and 2009 had, on average, almost 70 percent of their design drawings releasable at the time of the review, which is a consistent upward trend since 2003. However, most designs are still not stable at this point. Of the 28 programs in our latest assessment that have held a system-level critical design review, only 8 reported having a stable design. Only 2 of the 5 programs that held a critical design review in 2009 had a stable design at that point. The 5 programs reported that, on average, 83 percent of the total expected drawings were releasable.

While the design knowledge of DOD programs at the system-level critical design review has increased since 2003, these programs are still not regularly demonstrating that these designs can meet performance requirements by testing integrated prototypes before the critical design review—a best practice. None of the 5 programs in our latest assessment that held their critical design review in 2009 and planned to test a prototype did so before the review. Of the 33 programs that reported that they either had tested or were going to test an early system prototype and provided a critical design review date, only 4 did so before their critical design review. The Weapon Systems Acquisition Reform Act of 2009 requires that DOD policy ensure that the acquisition strategy for each major defense acquisition program provides for competitive prototypes before milestone B approval, unless a waiver is properly granted. This requirement should increase the percentage of programs demonstrating that the system’s design works as intended before the critical design review.

Some programs are taking steps to bring critical manufacturing processes into control, however many programs still rely on “after the fact” metrics. Capturing critical manufacturing knowledge before entering production helps ensure that a weapon system will work as intended and can be manufactured efficiently to meet cost, schedule, and quality targets. Identifying key product characteristics and the associated critical manufacturing processes is a key initial step to ensuring production elements are stable and in control. Seven programs in our latest assessment have identified their critical manufacturing processes, including four of the programs that entered production in 2009. Three of those seven programs reported that their critical manufacturing processes were in control. It is generally less costly—in terms of time and money—to eliminate product variation by controlling manufacturing processes than to perform extensive inspection after a product is built. However, many DOD programs rely on inspecting produced components instead of using statistical process control data in order to assess the maturity of their production processes. For example, 12 programs in our assessment reported tracking defects in delivered units, nonconformances, or scrap/rework as a way to measure production process maturity. The use of “after the fact” metrics is a reactive approach towards managing manufacturing quality as opposed to a prevention-based approach.

Programs are still not regularly testing production representative prototypes before committing to production. We have previously reported that in addition to demonstrating that the system can be built efficiently, production and postproduction costs are minimized when a fully integrated, capable prototype is demonstrated to show that the system will work as intended and in a reliable manner. The benefits of testing are maximized when the tests are completed prior to a production decision because making design changes after production begins can be both costly and inefficient. However, of the 32 programs in our assessment that could have tested a prototype before production, only 17 either tested or expect to test a fully configured, integrated, production-representative prototype before holding their production decision. In December 2008, DOD changed its policy to require programs to test production- representative articles before entering production.

More programs are using reliability growth curves before beginning production. Reliability growth testing provides visibility over how reliability is improving and uncovers design problems so fixes can be incorporated before production begins. According to DOD’s acquisition policy, a major defense acquisition program may not proceed beyond low-rate initial production until it has demonstrated acceptable reliability. Over half—22 of 40 programs that responded to our questionnaire—reported that they use a reliability growth curve, with 18 of these programs reporting they are currently meeting their established goals. In addition, 12 of 19 programs that expect to hold their production decision in 2010 and beyond reported using reliability growth curves and most stated they are currently meeting their goals. This practice should help these programs begin production with a reliable product design.

Observations on Other Factors That Can Affect Program Execution

Our 2010 assessment of weapon programs also included three observations on other areas related to DOD’s management of its weapons programs, including requirements, software management, and program office staffing. We have previously identified requirements changes and increases in software lines of code as sources of program instability that can contribute to cost growth and schedule delays. We have also reported that workforce challenges can hinder program execution and negatively affect program management and oversight.

A majority of programs changed key systems requirements after development start. Of the 42 programs in our 2010 assessment that reported tracking requirements changes, 23 programs reported having had at least one change (addition, reduction, enhancement, or deferment) to a key performance parameter—a top-level requirement—since development start. Further, nine programs experienced at least one change to a key system attribute—a lower level, but still a crucial requirement of the system. Eight programs reported major effects on the program as a result of these requirements changes, such as not meeting acquisition program baseline cost, schedule, and performance thresholds. DOD’s revised December 2008 acquisition policy attempts to reduce potentially disruptive requirements changes by requiring programs to hold annual configuration steering board meetings to ensure that significant technical changes are not approved without considering their effect on cost and schedule.

Many programs are at risk for cost growth and schedule delays because of software development issues. Seventeen of the 28 programs in our 2010 assessment that reported data on software lines of code estimated that the number of lines of code required for the system to function has grown or will grow by 25 percent or more—a predictor of future cost and schedule growth. Overall, the average growth or expected growth in lines of code for the 28 programs was about 92 percent. In addition to measuring growth in software lines of code, we have previously reported that collecting earned value management data for software development and tracking and containing software defects in phase are good management practices. Overall, 30 programs in our assessment reported collecting earned value management data to help manage software development. Thirty- two programs in our latest assessment also reported collecting some type of software defect data. For the 22 programs that responded a more specific question about defect correction, on average, only 69 percent of the defects were corrected in the phase of software development in which they occurred. Capturing software defects in phase is important because discovering defects out of phase can cause expensive rework later in programs.

Programs’ reliance on nongovernment personnel continues to increase in order to make up for shortfalls in government personnel and capabilities. In recent years, Congress and DOD have taken steps to ensure the acquisition workforce has the capacity, personnel, and skills needed to properly perform its mission; however, programs continue to struggle to fill all staff positions authorized. Only 19 of the 50 programs in our 2010 assessment that responded to our questions on staffing were able to fill all the positions they had been authorized. A commonly cited reason for not being able to fill positions was difficulty finding qualified candidates. As a result of staff shortfalls, program offices reported that program management and oversight has been degraded, contracting activities have been delayed, and program management costs have increased as contractors are used to fill the gap. Overall, 43 programs or 86 percent of those providing data reported utilizing support contractors to make up for shortfalls in government personnel and capabilities.

In addition, for the first time since we began reporting on program office staffing in 2008, programs reported having more nongovernment than government staff working in program offices (see table 2). The greatest numbers of support contractors are in engineering and technical positions, but their participation has increased in all areas, from program management and contracting to administrative support and other business functions.

Observations about DOD’s Implementation of Acquisition Reforms

DOD has begun to incorporate acquisition reforms into the acquisition strategies for new programs. Both DOD’s December 2008 acquisition policy revisions and the Weapon Systems Acquisition Reform Act of 2009 require programs to invest more time and resources in the front end of the acquisition process—refining concepts through early systems engineering, developing technologies, and building prototypes before starting system development. In addition, DOD policy requires establishment of configuration steering boards that meet annually to review all program requirements changes as well as to make recommendations on proposed descoping options that could help keep a program within its established cost and schedule targets. These steps could provide a foundation for establishing sound, knowledge-based business cases for individual weapon programs and are consistent with many of our past recommendations; however, if reform is to succeed and weapon program outcomes are to improve, they must continue to be reinforced in practice through decisions on individual programs.

Our analysis of the programs in our 2010 assessment allowed us to make two observations about the extent to which DOD is implementing recent acquisition reforms: Most of the ten programs in our 2010 assessment that had not yet entered system development reported having acquisitions strategies consistent with both DOD’s revised acquisition policy and the provisions of the Weapon Systems Acquisition Reform Act of 2009. Specifically, 8 programs in our assessment planned to develop competitive prototypes before milestone B. In addition, 7 programs have already scheduled a preliminary design review before milestone B.

Only a few programs reported holding configuration steering boards to review requirements changes, significant technical changes, or de-scoping options in 2009. Seven programs in our assessment reported holding configuration steering boards in 2009. Under DOD’s revised acquisition policy, ongoing acquisition category I and IA programs in development are required to conduct annual configuration steering boards to review requirements changes and significant technical configuration changes that have the potential to result in cost and schedule effects on the program. In addition, the program manager is expected to present de-scoping options to the board that could reduce program costs or moderate requirements. None of the programs reported that the boards that were held approved requirements changes or significant technical changes. One program—the P-8A Poseidon—reported that it presented de-scoping options to decrease cost and schedule risk on the program and had those options approved.

Concluding Observations on the Challenges to Achieving Lasting Reform

I would like to offer a few thoughts about other factors that should be considered so that we make the most out of today’s opportunity for meaningful change. First, I think it is useful to think of the processes that affect weapon system outcomes (requirements, funding, and acquisition) as being in a state of equilibrium. Poor outcomes—delays, cost growth, and reduced quantities—have been persistent for decades. If we think of these processes as merely “broken”, then some targeted repairs should fix them. I think the challenge is greater than that. If we think of these processes as being in equilibrium, where their inefficiencies are implicitly accepted as the cost of doing business, then the challenge for getting better outcomes is greater. Seen in this light, it will take considerable and sustained effort to change the incentives and inertia that reinforce the status quo.

Second, while actions taken and proposed by DOD and Congress are constructive and will serve to improve acquisition outcomes, one has to ask the question why extraordinary actions are needed to force practices that should occur normally. The answer to this question will shed light on the cultural or environmental forces that operate against sound management practices. For reforms to work, they will have to address these forces as well. For example, there have been a number of changes to make cost estimates more rigorous and realistic, but do these address all of the reasons why estimates are not already realistic? Clearly, more independence, methodological rigor, and better information about risk areas like technology will make estimates more realistic. On the other hand, realism is compromised as the competition for funding encourages programs to appear affordable. Also, when program sponsors present a program as more than a weapon system, but rather as essential to new fighting concepts, pressures exist to accept less than rigorous cost estimates. Reform must recognize and counteract these pressures as well.

Third, decisions on individual systems must reinforce good practices. Programs that have pursued risky and unexecutable acquisition strategies have succeeded in winning approval and funding. If reform is to succeed, then programs that present realistic strategies and resource estimates must succeed in winning approval and funding. Those programs that continue past practices of pushing unexecutable strategies must be denied funding before they begin. This will require sustained leadership from the Secretary of Defense, the Under Secretary of Defense for Acquisition, Technology and Logistics, and the military services, and the cooperation and support of Congress.

Fourth, consideration should be given to setting some limits on what is a reasonable length of time for developing a system. For example, if a program has to complete development within 5 or 6 years, this could serve as a basis to constrain requirements and exotic programs. It would also serve to get capability in the hands of the warfighter sooner.

Fifth, the institutional resources we have must match the outcomes we desire. For example, if more work must be done to reduce technical risk before development start—milestone B—DOD needs to have the organizational, people, and financial resources to do so. Once a program is approved for development, program offices and testing organizations must have the workforce with the requisite skills to manage and oversee the effort. Contracting instruments must be used that match the needs of the acquisition and protect the government’s interests. Finally, DOD must be judicious and consistent in how it relies on contractors.

Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions you or other Members of the Subcommittee may have at this time.

Contacts and Acknowledgements

For further information about this statement, please contact Michael J. Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Ronald E. Schwenn, Assistant Director, Kristine R. Hassinger, Carol T. Mebane, and Kenneth E. Patton.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.