Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon Program Outcomes
Released: 2008-09-25T00:00:00

Letter

DOD’s Major Acquisition Programs Continue to Experience Significant Cost Growth and Schedule Delays

DOD is not receiving expected returns on its large investment in weapon systems. The total acquisition cost of DOD’s 2007 portfolio of major programs under development or in production has grown by nearly $300 billion over initial estimates. While DOD is committing substantially more investment dollars to develop and procure new weapon systems, our analysis shows that the 2007 portfolio is experiencing greater cost growth and schedule delays than the fiscal years 2000 and 2005 portfolios (see table 1). Total acquisition costs for programs in DOD’s fiscal year 2007 portfolio have increased 26 percent from first estimates—compared to a 6-percent increase for programs in its fiscal year 2000 portfolio. Total RDT&E costs for programs in 2007 have increased by 40 percent from first estimates, compared to 27 percent for programs in 2000. The story is no better when expressed in unit costs. Schedule delays also continue to impact programs. On average, the current portfolio of programs has experienced a 21-month delay in delivering initial operational capability to the warfighter, and 14 percent are more than 4 years late.

Continued cost growth results in less funding being available for other DOD priorities and programs, while continued failure to deliver weapon systems on time delays providing critical capabilities to the warfighter. Put simply, cost growth reduces DOD’s buying power. As program costs increase, DOD must request more funding to cover the overruns, make trade-offs with existing programs, delay the start of new programs, or take funds from other accounts. Delays in providing capabilities to the warfighter result in the need to operate costly legacy systems longer than expected, find alternatives to fill capability gaps, or go without the capability. The warfighter’s urgent need for the new weapon system is often cited when the case is first made for developing and producing the system. However, DOD has already missed fielding dates for many programs and many others are behind schedule.

Fragmented Processes, Unexecutable Business Cases, and Lack of Knowledge Underlie Poor Acquisition Outcomes

Over the past several years our work has highlighted a number of underlying systemic causes for cost growth and schedule delays both at the strategic and at the program level. At the strategic level, DOD’s processes for identifying warfighter needs, allocating resources, and developing and procuring weapon systems—which together define DOD’s overall weapon system investment strategy—are fragmented and broken. At the program level, the military services propose and DOD approves programs without adequate knowledge about requirements and the resources needed to successfully execute the program within cost, schedule, and performance targets.

Key Acquisition Support Processes Are Fragmented and Result in Unsound Programs

DOD largely continues to define war fighting needs and make investment decisions on a service-by-service basis, and assess these requirements and their funding implications under separate decision-making processes. While DOD’s requirements process provides a framework for reviewing and validating needs, it does not adequately prioritize those needs and is not agile enough to meet changing warfighter demands. Ultimately, the process produces more demand for new programs than available resources can support. This imbalance promotes an unhealthy competition for funds that encourages programs to pursue overly ambitious capabilities, develop unrealistically low cost estimates and optimistic schedules, and to suppress bad news. Similarly, DOD’s funding process does not produce an accurate picture of the department’s future resource needs for individual programs—in large part because it allows programs to go forward with unreliable cost estimates and lengthy development cycles—not a sound basis for allocating resources and ensuring program stability. Invariably, DOD and the Congress end up continually shifting funds to and from programs—undermining well-performing programs to pay for poorly performing ones.

Initiating Programs with Unexecutable Business Cases Sets Them Up to Fail

At the program level, the key cause of poor outcomes is the consistent lack of disciplined analysis that would provide an understanding of what it would take to field a weapon system before system development. Our body of work in best practices has found that an executable business case is one that provides demonstrated evidence that (1) the identified needs are real and necessary and that they can best be met with the chosen concept and (2) the chosen concept can be developed and produced within existing resources—including technologies, funding, time, and management capacity. Although DOD has taken steps to revise its acquisition policies and guidance to reflect the benefits of a knowledge- based approach, we have found no evidence of widespread adoption of such an approach in the department. Our most recent assessment of major weapon systems found that the vast majority of programs began development with unexecutable business cases, and did not attain, or plan to achieve, adequate levels of knowledge before reaching design review and production start—the two key junctures in the process following development start (see figure 2).

Knowledge gaps are largely the result of a lack of disciplined systems engineering analysis prior to beginning system development. Systems engineering translates customer needs into specific product requirements for which requisite technological, software, engineering, and production capabilities can be identified through requirements analysis, design, and testing. Early systems engineering provides knowledge that enables a developer to identify and resolve gaps before product development begins. Because the government often does not perform the proper up-front analysis to determine whether its needs can be met, significant contract cost increases can occur as the scope of the requirements change or become better understood by the government and contractor. Not only does DOD not typically conduct disciplined systems engineering prior to beginning system development, it has allowed new requirements to be added well into the acquisition cycle. The acquisition environment encourages launching ambitious product developments that embody more technical unknowns and less knowledge about the performance and production risks they entail. A new weapon system is not likely to be approved unless it promises the best capability and appears affordable within forecasted available funding levels. We have recently reported on the negative impact that poor systems engineering practices have had on several programs such as the Global Hawk Unmanned Aircraft System, F-22A, Expeditionary Fighting Vehicle, Joint Air-to-Surface Standoff Missile and others.

With high levels of uncertainty about technologies, design, and requirements, program cost estimates and related funding needs are often understated, effectively setting programs up for failure. We recently compared the service and independent cost estimates for 20 major weapon system programs and found that the independent estimate was higher in nearly every case, but the difference between the estimates was typically not significant. We also found that both estimates were too low in most cases, and the knowledge needed to develop realistic cost estimates was often lacking. For example, program Cost Analysis Requirements Description documents—used to build the program cost estimate—are not typically based on demonstrated knowledge and therefore provide a shaky foundation for estimating costs. Cost estimates have proven to be off by billions of dollars in some of the programs we reviewed. For example, the initial Cost Analysis Improvement Group estimate for the Expeditionary Fighting Vehicle program was about $1.4 billion compared to a service estimate of about $1.1 billion, but development costs for the system are now expected to be close to $3.6 billion. Estimates this far off the mark do not provide the necessary foundation for sufficient funding commitments and realistic long-term planning.

When DOD consistently allows unsound, unexecutable programs to pass through the requirements, funding, and acquisition processes, accountability suffers. Program managers cannot be held accountable when the programs they are handed already have a low probability of success. In addition, they are not empowered to make go or no-go decisions, have little control over funding, cannot veto new requirements, and they have little authority over staffing. At the same time, program managers frequently change during a program’s development.

Limiting the length of development cycles would make it easier to more accurately estimate costs, predict the future funding needs, effectively allocate resources, and hold decision makers accountable. We have consistently emphasized the need for DOD’s weapon programs to establish shorter development cycles. DOD’s conventional acquisition process often requires as many as 10 or 15 years to get from program start to production. Such lengthy cycle times promote program instability—especially when considering DOD’s tendency to change requirements and funding as well as leadership. Constraining cycle times to 5 or 6 years would force programs to conduct more detailed systems engineering analyses, lend itself to fully funding programs to completion, and thereby increasing the likelihood that their requirements can be met within established time frames and available resources. An assessment of DOD’s acquisition system commissioned by the Deputy Secretary of Defense in 2006 similarly found that programs should be time-constrained to reduce pressure on investment accounts and increase funding stability for all programs.

The Way Forward: Potential Solutions

Our work shows that acquisition problems will likely persist until DOD provides a better foundation for buying the right things, the right way. This involves (1) maintaining the right mix of programs to invest in by making better decisions as to which programs should be pursued given existing and expected funding and, more importantly, deciding which programs should not be pursued; (2) ensuring that programs that are started can be executed by matching requirements with resources and locking in those requirements; and (3) making it clear that programs will then be executed based on knowledge and holding program managers responsible for that execution. We have made similar recommendations in past GAO reports, but DOD has disagreed with some and not fully implemented others.

These changes will not be easy to make. They will require DOD to reexamine not only its acquisition process, but its requirement setting and funding processes as well. They will also require DOD to change how it views program success, and what is necessary to achieve success. This includes changing the environment and incentives that lead DOD and the military services to overpromise on capability and underestimate costs in order to sell new programs and capture the funding needed to start and sustain them. Finally, none of this will be achieved without a true partnership among the department, the military services, the Congress, and the defense industry. All of us must embrace the idea of change and work diligently to implement it.

Buy the Right Things: Develop and Implement an Investment Strategy

The first, and most important, step toward improving acquisition outcomes is implementing a new DOD-wide investment strategy for weapon systems. We have reported that DOD should develop an overarching strategy and decision-making processes that prioritize programs based on a balanced match between customer needs and available department resources---that is the dollars, technologies, time, and people needed to achieve these capabilities. We also recommended that capabilities not designated as a priority should be set out separately as desirable but not funded unless resources were both available and sustainable. This means that the decision makers responsible for weapon system requirements, funding, and acquisition execution must establish an investment strategy in concert.

DOD’s Under Secretary of Defense for Acquisition, Technology and Logistics—DOD’s corporate leader for acquisition—should develop this strategy in concert with other senior leaders, for example, combatant commanders who would provide input on user needs; DOD’s comptroller and science and technology leaders, who would provide input on available resources; and acquisition executives from the military services, who could propose solutions. Finally, once priority decisions are made, Congress will need to enforce discipline through its legislative and oversight mechanisms.

Buy the Right Way: Ensure Individual Programs Can Be Executed

Once DOD has prioritized capabilities, it should work vigorously to make sure each new program can be executed before the acquisition begins. More specifically, this means assuring requirements for specific weapon systems are clearly defined and achievable given available resources and that all alternatives have been considered. System requirements should be agreed to by service acquisition executives as well as combatant commanders. Once programs begin, requirements should not change without assessing their potential disruption to the program and assuring that they can be accommodated within time and funding constraints. In addition, DOD should prove that technologies can work as intended before including them in acquisition programs. More ambitious technology development efforts should be assigned to the science and technology community until they are ready to be added to future generations of the product. DOD should also require the use of independent cost estimates as a basis for budgeting funds. Our work over the past 10 years has consistently shown when these basic steps are taken, programs are better positioned to be executed within cost and schedule.

To keep programs executable, DOD should demand that all milestone decisions be based on quantifiable data and demonstrated knowledge. These data should cover critical program facets such as cost, schedule, technology readiness, design readiness, production readiness, and relationships with suppliers. Development should not be allowed to proceed until certain knowledge thresholds are met—for example, a high percentage of engineering drawings completed at critical design review. DOD’s current policies encourage these sorts of metrics to be used as a basis for decision making, but they do not demand it. DOD should also place boundaries on the time allowed for system development.

To further ensure that programs can be executed, DOD should pursue an evolutionary path toward meeting user needs rather than attempting to satisfy all needs in a single step. This approach has been consistently used by successful commercial companies we have visited over the past decade because it provides program managers with more achievable requirements, which, in turn, facilitate shorter cycle times. With shorter cycle times, the companies we have studied have also been able to assure that program managers and senior leaders stay with programs throughout the duration of a program.

DOD has policies that encourage evolutionary development, but programs often favor pursuing more revolutionary, exotic solutions that will attract funds and support. The department and, more importantly, the military services, tend to view success as capturing the funding needed to start and sustain a development program. In order to do this, they must overpromise capability and underestimate cost. In order for DOD to move forward, this view of success must change. World-class commercial firms identify success as developing products within cost estimates and delivering them on time in order to survive in the marketplace. This forces incremental, knowledge-based product development programs that improve capability as new technologies are matured.

Hold People Accountable

To strengthen accountability, DOD must also clearly delineate responsibilities among those who have a role in deciding what to buy as well as those who have role in executing, revising, and terminating programs. Within this context, rewards and incentives must be altered so that success can be viewed as delivering needed capability at the right price and the right time, rather than attracting and retaining support for numerous new and ongoing programs.

To enable accountability to be exercised at the program level once a program begins, DOD will need to (1) match program manager tenure with development or the delivery of a product; (2) tailor career paths and performance management systems to incentivize longer tenures; (3) strengthen training and career paths as needed to ensure program managers have the right qualifications to manage the programs they are assigned to; (4) empower program managers to execute their programs, including an examination of whether and how much additional authority can be provided over funding, staffing, and approving requirements proposed after the start of a program; and (5) develop and provide automated tools to enhance management and oversight as well as to reduce the time required to prepare status information.

DOD also should hold contractors accountable for results. As we have recommended, this means structuring contracts so that incentives actually motivate contractors to achieve desired acquisition outcomes and withholding fees when those goals are not met.

Recent Congressional Initiatives and DOD Actions Aim to Promote a More Disciplined, Knowledge-Based Acquisition Approach

Recognizing the need for more discipline and accountability in the acquisition process, Congress recently enacted legislation that, if followed, could result in a better chance to spend resources wisely. Likewise, DOD has recently begun to develop several initiatives, based in part on congressional direction and GAO recommendations that, if implemented properly, could also provide a foundation for establishing a well balanced investment strategy and sound, knowledge-based business cases for individual acquisition programs.

Legislation Could Have a Positive Impact on Acquisition Outcomes

Congress has enacted legislation that requires DOD to take certain actions which, if followed, could instill more discipline into the front-end of the acquisition process when key knowledge is gained and ultimately improve acquisition outcomes. For example, legislation enacted in 2006 and 2008 requires decision-makers to certify that specific levels of knowledge have been demonstrated at key decision points early in the acquisition process before programs can receive milestone approval for the technology development phase or the system development phase respectively. The 2006 legislation also requires programs to track unit cost growth against their original baseline estimates—and not only their most recent estimates—and requires an additional assessment of the program if certain cost growth thresholds are reached. Other key legislation requires DOD to report on the department’s strategies for balancing the allocation of funds and other resources among major defense acquisition programs, and to identify strategies for enhancing the role of program managers in carrying out acquisition programs.

Recent DOD Actions Provide Opportunities for Improvement

DOD has also initiated actions aimed at improving investment decisions and weapon system acquisition outcomes, based in part on congressional direction and GAO recommendations. Each of the initiatives is designed to enable more informed decisions by key department leaders well ahead of a program’s start, decisions that provide a closer match between each program’s requirements and the department’s resources. For example: DOD is experimenting with a new concept decision review, different acquisition approaches according to expected fielding times, and panels to review weapon system configuration changes that could adversely affect program cost and schedule.

DOD is also testing portfolio management approaches in selected capability areas to facilitate more strategic choices about how to allocate resources across programs and also testing the use of capital budgeting as a potential means to stabilize program funding.

In September 2007, the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics issued a policy memorandum to ensure weapons acquisition programs were able to demonstrate key knowledge elements that could inform future development and budget decisions. This policy directed pending and future programs to include acquisition strategies and funding that provide for contractors to develop technically mature prototypes prior to initiating system development, with the hope of reducing technical risk, validating designs and cost estimates, evaluating manufacturing processes, and refining requirements.

DOD also plans to implement new practices that reflect past GAO recommendations intended to provide program managers more incentives, support, and stability. The department acknowledges that any actions taken to improve accountability must be based on a foundation whereby program managers can launch and manage programs toward greater performance, rather than focusing on maintaining support and funding for individual programs. DOD acquisition leaders have told us that any improvements to program managers’ performance hinge on the success of these departmental initiatives.

In addition, DOD has taken actions to strengthen the link between award and incentive fees with desired program outcomes, which has the potential to increase the accountability of DOD programs for fees paid and of contractors for results achieved.

If adopted and implemented properly these actions could provide a foundation for establishing sound, knowledge-based business cases for individual acquisition programs, and the means for executing those programs within established cost, schedule, and performance goals.

Concluding Observations on Achieving Successful and Lasting Reform

DOD understands what it needs to do at the strategic and at the program level to improve acquisition outcomes. The strategic vision of the current Under Secretary of Defense for Acquisition, Technology and Logistics acknowledges the need to create a high-performing, boundary-less organization—one that seeks out new ideas and new ways of doing business and is prepared to question requirements and traditional processes. Past efforts have had similar goals, yet we continue to find all too often that DOD’s investment decisions are too service- and program- centric and that the military services overpromise capabilities and underestimate costs to capture the funding needed to start and sustain development programs. This acquisition environment has been characterized in many different ways. For example, some have described it as a “conspiracy of hope,” in which industry is encouraged to propose unrealistic cost estimates, optimistic performance, and understated technical risks during the proposal process and DOD is encouraged to accept these proposals as the foundation for new programs. Either way, it is clear that DOD’s implied definition of success is to attract funds for new programs and to keep funds for ongoing programs, no matter what the impact. DOD and the military services cannot continue to view success through this prism.

More legislation can be enacted and policies can be written, but until DOD begins making better choices that reflect joint capability needs and matches requirements with resources, the acquisition environment will continue to produce poor outcomes. It should not be necessary to take extraordinary steps to ensure needed capabilities are delivered to the warfighter on time and within costs. Executable programs should be the natural outgrowth of a disciplined, knowledge-based process. While DOD’s current policy supports a knowledge-based, evolutionary approach to acquiring new weapons, in practice decisions made on individual programs often sacrifice knowledge and realism in favor of revolutionary solutions. Meaningful and lasting reform will not be achieved until DOD changes the acquisition environment and the incentives that drive the behavior of DOD decision-makers, the military services, program managers, and the defense industry. Finally, no real reform can be achieved without a true partnership among all these players and the Congress.

Mr. Chairman, this concludes my prepared statement. I would be happy to answer any questions you may have at this time.

Contacts and Staff Acknowledgements

For further questions about this statement, please contact Michael J. Sullivan at (202) 512-4841. Individuals making key contributions to this statement include Ron Schwenn, Assistant Director; Kenneth E. Patton, and Alyssa B. Weir.

Related GAO Products

Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes. GAO-08-619. Washington, D.C.: July 2, 2008.

Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. GAO-08-782T. Washington, D.C.: June 3, 2008.

Defense Acquisitions: Results of Annual Assessment of DOD Weapon Programs. GAO-08-674T. Washington, D.C.: April 29, 2008.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-08-467SP. Washington, D.C.: March 31, 2008.

Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD’s Acquisition Environment and Weapon System Quality. GAO-08-294. Washington, D.C.: Feb. 1, 2008.

Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs. GAO-07-1134SP, Washington, D.C.: July 2007.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-07-406SP. Washington, D.C.: March 30, 2007.

Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD’s Acquisition Outcomes. GAO-07-388, Washington, D.C.: March 30, 2007.

Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1, 2005.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004.

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisitions: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.