Defense Acquisitions: Results of Annual Assessment of DOD Weapon Programs
Released: 2008-04-29T00:00:00

Letter

DOD’s Major Acquisition Programs Continue to Experience Significant Cost Growth and Schedule Delays

DOD is not receiving expected returns on its large investment in weapon systems. While it is committing substantially more investment dollars to develop and procure new weapon systems, our analysis shows that the 2007 portfolio of major defense acquisition programs is experiencing greater cost growth and schedule delays than programs in fiscal years 2000 and 2005. For example, as shown in table 1, total acquisition costs for 2007 programs have increased 26 percent from first estimates, whereas programs in fiscal year 2000 had increased by 6 percent. Total RDT&E costs for programs in 2007 have increased by 40 percent from first estimates, compared to 27 percent for programs in 2000. The story is no better when expressed in unit costs. Based on our analysis for the 2007 portfolio, 44 percent of DOD’s major defense acquisition programs are paying at least 25 percent more per unit than originally expected. The percentage of programs experiencing a 25 percent or more increase in program acquisition unit costs in fiscal year 2000 was 37 percent.

The consequence of cost growth is reduced buying power, which can represent significant opportunity costs for DOD. In other words, every dollar spent on inefficiencies in acquiring one weapon system is less money available for other priorities and programs. Total acquisition cost for the current portfolio of major programs under development or in production has grown by nearly $300 billion over initial estimates. As program costs increase, DOD must request more funding to cover the overruns, make trade-offs with existing programs, delay the start of new programs, or take funds from other accounts.

Just as importantly, DOD has already missed fielding dates for many programs and many others are behind schedule. Because of program delays, warfighters often have to operate costly legacy systems longer than expected, find alternatives to fill capability gaps, or go without the capability. The warfighter’s urgent need for the new weapon system is often cited when the case is first made for developing and producing the system. However, on average, the current portfolio of programs has experienced a 21-month delay in delivering initial operational capability to the warfighter and, in fact, 14 percent are more than 4 years late.

DOD Weapon System Programs Continue to Move Forward Without Proper Knowledge about Requirements, Technology, Design, and Manufacturing Processes

In assessing the 72 weapon programs, we found no evidence of widespread adoption of a knowledge-based acquisition process within DOD despite polices to the contrary. Reconciling this discrepancy between policy and practice is essential for getting better outcomes for DOD programs. The majority of programs in our assessment this year proceeded with lower levels of knowledge at critical junctures and attained key elements of product knowledge later in development than expected under best practices (see fig. 1). This exposes programs to significant and unnecessary technology, design, and production risks, and ultimately leads to cost growth and schedule delays. The building of knowledge over a product’s development is cumulative, as one knowledge point builds on the next, and failure to capture key product knowledge can lead to problems that eventually cascade and become magnified throughout product development and production.

Programs Begin without Matching Product Requirements with Available Resources

Very few of the programs we assessed started system development with evidence that the proposed solution was based on mature technologies and proven design features. As a result, programs are still working to mature technologies during system development and production, which causes significantly higher cost growth than programs that start development with mature technologies. Only 12 percent of the programs in our assessment demonstrated all of their critical technologies as fully mature at the start of system development and they have had much better outcomes than the others. For those programs in our assessment with immature technologies at development start, total RDT&E costs grew by 44 percent more than for programs that began with mature technologies. More often than not, programs were still maturing technologies late into development and even into production.

In addition to ensuring that technologies are mature, best practices for product development suggest that the developer should have delivered a preliminary design of the proposed weapon system based on a robust systems engineering process before committing to system development. This process should allow the developer—the contractor responsible for designing the weapon system—to analyze the customer’s expectations for the product and identify gaps between resources and those expectations, which then can be addressed through additional investments, alternate designs, and ultimately trade-offs. Only ten percent of the programs in our assessment had completed their preliminary design review prior to committing to system development. The other 90 percent averaged about 2 1/2 years into system development before the review was completed or planned to be completed. Programs like the Aerial Common Sensor and Joint Strike Fighter did not deliver a sound preliminary design at system development start and discovered problems early in their design activities that required substantial resources be added to the programs or, in the case of Aerial Common Sensor, termination of the system development contract.

Programs Continue to Move into System Demonstration and Production without Achieving Design Stability

Knowing that a product’s design is stable before system demonstration reduces the risk of costly design changes occurring during the manufacturing of production representative prototypes—when investments in acquisitions become much more significant. Only a small portion of the programs in our assessment that have held a design review captured the necessary knowledge to ensure that they had mature technologies at system development start and a stable system design before entering the more costly system demonstration phase of development. Over half of the programs in our assessment did not even have mature technologies at the design review (knowledge that actually should have been achieved before system development start). Also, less than one-quarter of the programs that provided data on drawings released at the design review reached the best practices standard of 90 percent. We have found that programs moving forward into system demonstration with low levels of design stability are more likely than other programs to encounter costly design changes and parts shortages that in turn caused labor inefficiencies, schedule delays, and quality problems. Even by the beginning of production, more than a third of the programs that had entered this phase still had not released 90 percent of their engineering drawings.

In addition, we found that over 80 percent of the programs providing data did not or did not plan to demonstrate the successful integration of the key subsystems and components needed for the product through an integration laboratory, or better yet, through testing an early system prototype by the design review. For example, the Navy’s E-2D Advanced Hawkeye moved past the design review and entered systems demonstration without fully proving—through the use of an integration lab or prototype—that the design could be successfully integrated. The program did not have all the components operational in a systems integration lab until almost 2 years after the design review. While the program estimated it had released 90 percent of the drawings needed for the system by the design review, as it was conducting system integration activities, it discovered that it needed substantially more drawings. This increase means that the program really had completed only 53 percent of the drawings prior to the review, making it difficult to ensure the design was stable.

Programs Enter Production without Demonstrating Acceptable Manufacturing Processes and Weapon System Performance

In addition to lacking mature technologies and design stability, most programs have not or do not plan to capture critical manufacturing and testing knowledge before entering production. This knowledge ensures that the product will work as intended and can be manufactured efficiently to meet cost, schedule, and quality targets. Of the 26 programs in our assessment that have had production decisions, none provided data showing that they had all their critical manufacturing processes in statistical control by the time they entered into the production phase. In fact, only 3 of these programs indicated that they had even identified the key product characteristics or associated critical manufacturing processes—key initial steps to ensuring critical production elements are stable and in control. Failing to capture key manufacturing knowledge before producing the product can lead to inefficiencies and quality problems. For example, the Wideband Global SATCOM program encountered cost and schedule delays because contractor personnel installed fasteners incorrectly. Discovery of the problem resulted in extensive inspection and rework to correct the deficiencies, contributing to a 15-month schedule delay.

In addition to demonstrating that the product can be built efficiently, our work has shown that production and post-production costs are minimized when a fully integrated, capable prototype is demonstrated to show it will work as intended and in a reliable manner. We found that many programs are susceptible to discovering costly problems late in development, when the more complex software and advanced capabilities are tested. Of the 33 programs that provided us data about the overlap between system development and production, almost three-quarters still had or planned to have system demonstration activities left to complete after production had begun. For 9 programs, the amount of system development work remaining was estimated to be over 4 years. This practice of beginning production before successfully demonstrating that the weapon system will work as intended increases the potential for discovering costly design changes that ripple through production into products already fielded.

Forty programs we assessed provided us information on when they had or planned to have tested a fully configured, integrated production representative article (i.e., prototype) in the intended environment. Of these, 62 percent reported that they did not conduct or do not plan to conduct that test before a production decision. We also found examples where product reliability is not being demonstrated in a timely fashion. Making design changes to achieve reliability requirements after production begins is inefficient and costly. For example, despite being more than 5 years past the production decision, the Air Force’s Joint Air-to-Surface Standoff Missile experienced four failures during four flight tests in 2007, resulting in an overall missile reliability rate of less than 60 percent. The failures halted procurement of new missiles by the Air Force until the problems could be resolved.

Absence of Disciplined Systems Engineering Practices Leads to Unexecutable Business Cases

DOD’s poor acquisition outcomes stem from the absence of knowledge that disciplined systems engineering practices can bring to decision makers prior to beginning a program. Systems engineering is a process that translates customer needs into specific product requirements for which requisite technological, software, engineering, and production capabilities can be identified. These activities include requirements analysis, design, and testing in order to ensure that the product’s requirements are achievable given available resources. Early systems engineering provides knowledge that enables a developer to identify and resolve gaps before product development begins. Consequently, establishing a sound acquisition program with an executable business case depends on determining achievable requirements based on systems engineering that are agreed to by both the acquirer and developer before a program’s initiation. We have recently reported on the impact that poor systems engineering practices have had on several programs such as the Global Hawk Unmanned Aircraft System, F-22A, Expeditionary Fighting Vehicle, Joint Air-to-Surface Standoff Missile and others.

When early systems engineering, specifically requirements analysis, is not performed, increased cost risk to the government and long development cycle times can be the result. DOD awards cost reimbursement type contracts for the development of major weapon systems because of the risk and uncertainty involved with its programs. Because the government often does not perform the necessary systems engineering analysis before a contract is signed to determine whether a match exists between requirements and available resources, significant contract cost increases can occur as the scope of the requirements change or becomes better understood by the government and contractor. Another potential consequence of the lack of requirements analysis is unpredictable cycle times. Requirements that are limited and well-understood contribute to shorter, more predictable cycle times. Long cycle times promote instability, especially considering DOD’s tendency to have changing requirements and program manager turnover. On the other hand, time- defined developments can allow for more frequent assimilation of new technologies into weapon systems and speed new capabilities to the warfighter. In fact, DOD itself suggests that system development should be limited to about 5 years.

Additional Factors Can Contribute to Poor Weapon Program Outcomes

This year, we gathered new data focused on other factors we believe could have a significant influence on DOD’s ability to improve cost and schedule outcomes. These factors were changes to requirements after development began, the length of program managers’ tenure, reliance on contractors for program support, and difficulty managing software development.

Foremost, several DOD programs in our assessment incurred requirement changes after the start of system development and experienced cost increases. Among the 46 programs we surveyed, RDT&E costs increased by 11 percent over initial estimates for programs that have not had requirements changes, while they increased 72 percent among those that had requirements changes (see fig. 2).

At the same time, DOD’s practice of frequently changing program managers during a program’s development makes it difficult to hold them accountable for the business cases that they are entrusted to manage and deliver. Our analysis indicates that for 39 major acquisition programs started since March 2001, the average time in system development was about 37 months. The average tenure for program managers on those programs during that time was about 17 months—less than half of what is required by DOD policy.

We also found that DOD is relying more on contractors to support the management and oversight of weapon system acquisitions and contracts. For 52 DOD programs that provided information, about 48 percent of the program office staff was composed of individuals outside of DOD (see table 2). In a prior review of space acquisition programs, we found that 8 of 13 cost-estimating organizations and program offices believed the number of cost estimators was inadequate and we found that 10 of those offices had more contractor personnel preparing cost estimates than government personnel. We also found examples during this year’s assessment where the program offices expressed concerns about having inadequate personnel to conduct their program office roles.

Finally, as programs rely more heavily on software to perform critical functions for weapon systems, we found that a large number of programs are encountering difficulties in managing their software development. Roughly half of the programs that provided us software data had at least a 25 percent growth in their expected lines of code—a key metric used by leading software developers—since system development started. For example, software requirements were not well understood on the Future Combat Systems when the program began; and as the program moves toward preliminary design activities, the number of lines of software code has nearly tripled. Changes to the lines of code needed can indicate potential cost and schedule problems.

The Way Forward: Potential Solutions

Our work shows that acquisition problems will likely persist until DOD provides a better foundation for buying the right things, the right way. This involves (1) maintaining the right mix of programs to invest in by making better decisions as to which programs should be pursued given existing and expected funding and, more importantly, deciding which programs should not be pursued; (2) ensuring that programs that are started are executable by matching requirements with resources and locking in those requirements; and (3) making it clear that programs will then be executed based on knowledge and holding program managers responsible for that execution. We have made similar recommendations in past GAO reports.

These changes will not be easy to make. They will require DOD to reexamine not only its acquisition process, but its requirement setting and funding processes as well. They will also require DOD to change how it views program success, and what is necessary to achieve success. This includes changing the environment and incentives that lead DOD and the military services to overpromise on capability and underestimate costs in order to sell new programs and capture the funding needed to start and sustain them. Finally, none of this will be achieved without a true partnership among the department, the military services, the Congress, and the defense industry. All of us must embrace the idea of change and work diligently to implement it.

Buy the Right Things: Develop and Implement an Investment Strategy

The first, and most important, step toward improving acquisition outcomes is implementing a new DOD-wide investment strategy for weapon systems. We have reported that DOD should develop an overarching strategy and decision-making processes that prioritize programs based on a balanced match between customer needs and available department resources---that is the dollars, technologies, time, and people needed to achieve these capabilities. We also recommended that capabilities not designated as a priority should be set out separately as desirable but not funded unless resources were both available and sustainable. This means that the decision makers responsible for weapon system requirements, funding, and acquisition execution must establish an investment strategy in concert.

DOD’s Under Secretary of Defense for Acquisition, Technology and Logistics—DOD’s corporate leader for acquisition—should develop this strategy in concert with other senior leaders, for example, combatant commanders who would provide input on user needs; DOD’s comptroller and science and technology leaders, who would provide input on available resources; and acquisition executives from the military services, who could propose solutions. Finally, once priority decisions are made, Congress will need to enforce discipline through its legislative and oversight mechanisms.

Buy the Right Way: Ensure Individual Programs Are Executable

Once DOD has prioritized capabilities, it should work vigorously to make sure each new program is executable before the acquisition begins. More specifically, this means assuring requirements for specific weapon systems are clearly defined and achievable given available resources and that all alternatives have been considered. System requirements should be agreed to by service acquisition executives as well as combatant commanders. Once programs begin, requirements should not change without assessing their potential disruption to the program and assuring that they can be accommodated within time and funding constraints. In addition, DOD should prove that technologies can work as intended before including them in acquisition programs. More ambitious technology development efforts should be assigned to the science and technology community until they are ready to be added to future generations of the product. DOD should also require the use of independent cost estimates as a basis for budgeting funds. Our work over the past 10 years has consistently shown when these basic steps are taken, programs are better positioned to be executed within cost and schedule.

To keep programs executable, DOD should demand that all go/no-go decisions be based on quantifiable data and demonstrated knowledge. These data should cover critical program facets such as cost, schedule, technology readiness, design readiness, production readiness, and relationships with suppliers. Development should not be allowed to proceed until certain knowledge thresholds are met—for example, a high percentage of engineering drawings completed at critical design review. DOD’s current policies encourage these sorts of metrics to be used as a basis for decision making, but they do not demand it. DOD should also place boundaries on the time allowed for system development.

To further ensure that programs are executable, DOD should pursue an evolutionary path toward meeting user needs rather than attempting to satisfy all needs in a single step. This approach has been consistently used by successful commercial companies we have visited over the past decade because it provides program managers with more achievable requirements, which, in turn, facilitate shorter cycle times. With shorter cycle times, the companies we have studied have also been able to assure that program managers and senior leaders stay with programs throughout the duration of a program.

DOD has policies that encourage evolutionary development, but programs often favor pursuing more revolutionary, exotic solutions that will attract funds and support. The department and, more importantly, the military services, tend to view success as capturing the funding needed to start and sustain a development program. In order to do this, they must overpromise capability and underestimate cost. In order for DOD to move forward, this view of success must change. World-class commercial firms identify success as developing products within cost estimates and delivering them on time in order to survive in the marketplace. This forces incremental, knowledge-based product development programs that improve capability as new technologies are matured.

Hold People Accountable

To strengthen accountability, DOD must also clearly delineate responsibilities among those who have a role in deciding what to buy as well as those who have role in executing, revising, and terminating programs. Within this context, rewards and incentives must be altered so that success can be viewed as delivering needed capability at the right price and the right time, rather than attracting and retaining support for numerous new and ongoing programs.

To enable accountability to be exercised at the program level once a program begins, DOD will need to (1) match program manager tenure with development or the delivery of a product; (2) tailor career paths and performance management systems to incentivize longer tenures; (3) strengthen training and career paths as needed to ensure program managers have the right qualifications for run the programs they are assigned to; (4) empower program managers to execute their programs, including an examination of whether and how much additional authority can be provided over funding, staffing, and approving requirements proposed after the start of a program; and (5) develop and provide automated tools to enhance management and oversight as well as to reduce the time required to prepare status information.

DOD also should hold contractors accountable for results. As we have recommended, this means structuring contracts so that incentives actually motivate contractors to achieve desired acquisition outcomes and withholding fees when those goals are not met.

Recent DOD Actions Provide Opportunities for Improvement

DOD has taken actions related to some of these steps. Based in part on GAO recommendations and congressional direction, DOD has recently begun to develop several initiatives that, if adopted and implemented properly, could provide a foundation for establishing sound, knowledge- based business cases for individual acquisition programs and improving program outcomes. For example, DOD is experimenting with a new concept decision review, different acquisition approaches according to expected fielding times, and panels to review weapon system configuration changes that could adversely affect program cost and schedule. In addition, in September 2007 the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics issued a policy memorandum to ensure weapon acquisition programs were able to demonstrate key knowledge elements that could inform future development and budget decisions. This policy directed pending and future programs to include acquisition strategies and funding that provide for two or more competing contractors to develop technically mature prototypes through system development start (knowledge point 1), with the hope of reducing technical risk, validating designs and cost estimates, evaluating manufacturing processes, and refining requirements. Each of the initiatives is designed to enable more informed decisions by key department leaders well ahead of a program’s start, decisions that provide a closer match between each program’s requirements and the department’s resources.

DOD also plans to implement new practices similar to past GAO recommendations that are intended to provide program managers more incentives, support, and stability. The department acknowledges that any actions taken to improve accountability must be based on a foundation whereby program managers can launch and manage programs toward greater performance, rather than focusing on maintaining support and funding for individual programs. DOD acquisition leaders have told us that any improvements to program managers’ performance hinge on the success of the department’s initiatives.

In addition, DOD has taken actions to strengthen the link between award and incentive fees with desired program outcomes, which has the potential to increase the accountability of DOD programs for fees paid and of contractors for results achieved.

Concluding Observations

In closing, the past year has seen several new proposed approaches to improve the way DOD buys weapons. These approaches have come from within the department, from highly credible commissions established by the department, and from GAO. They are based on solid principles. If they are to produce better results, however, they must heed the lessons taught—but perhaps not learned—by various past studies and by DOD’s acquisition history itself. Specifically, DOD must do a better job of prioritizing its needs in the context of the nation’s greater fiscal challenges. It must become more disciplined in managing the mix of programs to meet available funds. If everything is a priority, nothing is a priority.

Policy must also be manifested in decisions on individual programs or reform will be blunted. DOD’s current acquisition policy is a case in point. The policy supports a knowledge-based, evolutionary approach to acquiring new weapons. However, the practice—decisions made on individual programs—sacrifices knowledge and realism about what can done within the available time and funding in favor of revolutionary solutions.

Reform will not be real unless each weapon system is shown to be both a worthwhile investment and a realistic, executable program based on the technology, time, and money available. This cannot be done until the acquisition environment is changed along with the incentives associated with it. DOD and the military services cannot continue to view success through the prism of securing the funding needed to start and sustain new programs. Success must be defined in terms of delivering the warfighter capabilities when needed and as promised and incentives must be aligned to encourage a disciplined, knowledge-based approach to achieve this end.

The upcoming change in administration presents challenges as well as opportunities to improve the process and its outcomes through sustained implementation of best practices, as well as addressing new issues that may emerge. Significant changes will only be possible with greater, and continued, department level support, including strong and consistent vision, direction, and advocacy from DOD leadership, as well as sustained oversight and cooperation from the Congress. In addition, all of the players involved with acquisitions—the requirements community; the Joint Chiefs of Staff; the comptroller; the Under Secretary of Defense for Acquisition, Technology and Logistics; and perhaps most importantly, the military services—must be unified in implementing reforms from top to bottom.

Mr. Chairmen and Members of the Committee and Subcommittee, this concludes my statement. I will be happy to take any questions that you may have at this time.

Contacts and Staff Acknowledgements

For further questions about this statement, please contact Michael J. Sullivan at (202) 512-4841. Individuals making key contributions to this statement include Ron Schwenn, Assistant Director; Ridge C. Bowman; Quindi C. Franco; Matthew B. Lea; Brian Mullins; Kenneth E. Patton, and Alyssa B. Weir.

Related GAO Products

Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD’s Acquisition Environment and Weapon System Quality. GAO-08-294. Washington, D.C.: Feb. 1, 2008.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO- 07-406SP. Washington, D.C.: March 30, 2007.

Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006.

Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1, 2005.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, D.C.: November 15, 2005.

Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004.

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003.

Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept Technology Demonstration. GAO-03-52. Washington, D.C.: December 2, 2002.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisitions: Improved Program Outcomes Are Possible. GAO/T- NSIAD-98-123. Washington, D.C.: March 17, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.