DOD Acquisition Outcomes: A Case for Change
Released: 2005-11-15T00:00:00

Letter

The Case for Change

The way DOD develops and produces its major weapons systems has had disappointing outcomes. There is a vast difference between DOD’s budgeting plans and the reality of the cost of its systems. Performance, if it is defined as the capability that actually reaches the warfighter, often falls short, as cost increases result in fewer quantities of produced systems and schedule slips. Performance, if it is defined as an acceptable return on investment, has not lived up to promises.

Table 1 illustrates seven programs with a significant reduction in buying power; we have reported similar outcomes in many more programs. For example, the Air Force initially planned to buy 648 F/A-22 Raptor tactical aircraft at a program acquisition unit cost of about $125 million (fiscal year 2006 dollars). Technology and design components matured late in the development of the aircraft, which contributed to cost growth and schedule delays. Now, the Air Force plans to buy 181 aircraft at a program acquisition unit cost of about $361 million, an almost 189 percent increase.

Furthermore, the conventional acquisition process is not agile enough for today’s demands. Congress has expressed concern that urgent warfighting requirements are not being met in the most expeditious manner and has put in place several authorities for rapid acquisition to work around the process. The U.S. Joint Forces Command’s Limited Acquisition Authority and the Secretary of Defense’s Rapid Acquisition Authority seek the ability to get warfighting capability to the field faster. According to U.S. Joint Forces Command officials, it is only through Limited Acquisition Authority that the command has the authority to satisfy the unanticipated, unbudgeted, urgent mission needs of other combatant commands. With a formal process that requires as many as 5, 10, or 15 years to get from program start to production, such experiments are needed to meet the warfighters’ needs.

Today we are at a crossroad. Our nation is on an unsustainable fiscal path. Long-term budget simulations by GAO, the Congressional Budget Office, and others show that, over the long term, we face a large and growing structural deficit due primarily to known demographic trends and rising health care costs. Continuing on this unsustainable fiscal path will gradually erode, if not suddenly damage, our economy, our standard of living, and ultimately our national security. Federal discretionary spending, along with other federal policies and programs, will face serious budget pressures in the coming years stemming from new budgetary demands and demographic trends. Defense spending falls within the discretionary spending accounts. Further, current military operations, such as those in Afghanistan and Iraq, consume a large share of DOD resources and are causing faster wear on existing weapons. Refurbishment or replacement sooner than planned is putting further pressure on DOD’s investment accounts.

At the same time DOD is facing these problems, programs are commanding larger budgets. DOD is undertaking new efforts that are expected to be the most expensive and complex ever and on which DOD is heavily relying to fundamentally transform military operations. And it is giving contractors increased program management responsibilities to develop requirements, design products, and select major system and subsystem contractors. Table 2 shows that just 5 years ago, the top five weapon systems cost about $291 billion combined; today, the top five weapon systems cost about $550 billion.

If these megasystems are managed with traditional margins of error, the financial consequences can be dire, especially in light of a constrained discretionary budget.

Success for acquisitions means making sound decisions to ensure that program investments are getting promised returns. In the commercial world, successful companies have no choice but to adopt processes and cultures that emphasize basing decisions on knowledge, reducing risks prior to undertaking new efforts, producing realistic cost and schedule estimates, and building-in quality in order to deliver products to customers at the right price, the right time, and the right cost. At first blush, it would seem DOD’s definition of success would be very similar: deliver capability to the warfighter at the right price, the right time, and the right cost. However, this is not happening within DOD. In an important sense, success has come to mean starting and continuing programs even when cost, schedule, and quantities must be sacrificed.

DOD knows what to do to improve acquisitions but finds it difficult to apply the controls or assign the accountability necessary for successful outcomes. To understand why these problems persist, we must look not just at the product development process but at the underlying requirements and budgeting processes to define problems and find solutions.

A Knowledge-Based Process Can Lead to Better Outcomes

Over the last several years, we have undertaken a body of work that examines weapon acquisition issues from a perspective that draws upon lessons learned from best product development practices. Leading commercial firms expect that their program managers will deliver high- quality products on time and within budget. Doing otherwise could result in the customer walking away. Thus, those firms have created an environment and adopted practices that put their program managers in a good position to succeed in meeting these expectations. Collectively, these practices comprise a process that is anchored in knowledge. It is a process in which technology development and product development are treated differently and managed separately. The process of developing technology culminates in discovery—the gathering of knowledge—and must, by its nature, allow room for unexpected results and delays. Leading firms do not ask their product managers to develop technology. Successful programs give responsibility for maturing technologies to a science and technology organizations, rather than the program or product development managers. The process of developing a product culminates in delivery, and, therefore, gives great weight to design and production. The firms demand—and receive—specific knowledge about a new product before production begins. A program does not go forward unless a strong business case on which the program was originally justified continues to hold true.

Successful product developers ensure a high level of knowledge is achieved at key junctures in development. We characterize these junctures as knowledge points. These knowledge points and associated indicators are defined as follows:

Knowledge point 1: Resources and needs match. This point occurs when a sound business case is made for the product—that is, a match is made between the customer’s requirements and the product developer’s available resources in terms of knowledge, time, money, and capacity. Achieving a high level of technology maturity at the start of system development is an important indicator of whether this match has been made. This means that the technologies needed to meet essential product requirements have been demonstrated to work in their intended environment.

Knowledge point 2: Product design is stable. This point occurs when a program determines that a product’s design is stable— that is, it will meet customer requirements, as well as cost, schedule and reliability targets. A best practice is to achieve design stability at the system-level critical design review, usually held midway through development. Completion of at least 90 percent of engineering drawings at the system design review provides tangible evidence that the design is stable.

Knowledge point 3: Production processes are mature. This point is achieved when it has been demonstrated that the company can manufacture the product within cost, schedule, and quality targets. A best practice is to ensure that all key manufacturing processes are in statistical control—that is, they are repeatable, sustainable, and capable of consistently producing parts within the product’s quality tolerances and standards—at the start of production.

A result of this knowledge-based process is evolutionary product development, an incremental approach that enables developers to rely more on available resources rather than making promises about unproven technologies. Predictability is a key to success as successful product developers know that invention cannot be scheduled and its cost is difficult to estimate. They do not bring technology into new product development unless that technology has been demonstrated to meet the user’s requirements. Allowing technology development to spill over into product development puts an extra burden on decision makers and provides a weak foundation for making product development estimates. While the user may not initially receive the ultimate capability under this approach, the initial product is available sooner and at a lower, more predictable cost.

There is a synergy in this process, as the attainment of each successive knowledge point builds on the preceding one. Metrics gauge when the requisite level of knowledge has been attained. Controls are used to attain a high level of knowledge before making additional significant investments. Controls are considered effective if they are backed by measurable criteria and if decision makers are required to consider them before deciding to advance a program to the next level. Effective controls help decision makers gauge progress in meeting cost, schedule, and performance goals and ensure that managers will (1) conduct activities to capture relevant product development knowledge, (2) provide evidence that knowledge was captured, and (3) hold decision reviews to determine that appropriate knowledge was captured to move to the next phase. The result is a product development process that holds decision makers accountable and delivers excellent results in a predictable manner.

A hallmark of an executable program is shorter development cycle times, which allow more systems to enter production more quickly. DOD itself suggests that product development should be limited to about 5 years. Time constraints, such as this, are important because they serve to limit the initial product’s requirements. Limiting product development cycle times to 5 years or less would allow for more frequent assimilation of new technologies into weapon systems, speeding new technology to the warfighter, hold program managers accountable, as well as make more frequent and predictable work in production, where contractors and the industrial base can profit by being efficient.

Despite Policy, DOD Is Not Employing a Knowledge-Based Process

DOD’s policy adopts the knowledge-based, evolutionary approach used by leading commercial companies that enables developers to rely more on available resources rather than making promises about unproven technologies. The policy provides a framework for developers to ask themselves at key decision points whether they have the knowledge they need to move to the next phase of acquisition. For example, DOD Directive 5000.1 states that program managers “shall provide knowledge about key aspects of a system at key points in the acquisition process,” such as demonstrating “technologies in a relevant environment … prior to program initiation.” This knowledge-based framework can help managers gain the confidence they need to make significant and sound investment decisions for major weapon systems. In placing greater emphasis on evolutionary product development, the policy sets up a more manageable environment for achieving knowledge.

However, the longstanding problem of programs beginning development with immature technologies is continuing to be seen on even the newest programs. Several programs approved to begin product development within only the last few years began with most of their technologies immature and have already experienced significant development cost increases. In the case of the Army’s Future Combat Systems, nearly 2 years after program launch and with $4.6 billion invested, only 1 out of more than 50 critical technologies is considered mature and the research and development cost estimate has grown by 48 percent.

In March 2005, we reported that very few programs—15 percent of the programs we assessed—began development having demonstrated high levels of technology maturity. Acquisition unit costs for programs leveraging mature technologies increased by less than 1 percent, whereas programs that started development with immature technologies experienced an average acquisition unit cost increase of nearly 21 percent over the first full estimate.

Establishing a Sound Business Case Depends on Disciplined Requirements and Funding Process

The decision to start a new program is the most highly leveraged point in the product development process. Establishing a sound business case for individual programs depends on disciplined requirements and funding processes. Our work has shown that DOD’s requirements process generates more demand for new programs than fiscal resources can support. DOD compounds the problem by approving so many highly complex and interdependent programs. Moreover, once a program is approved, requirements can be added along the way that increases costs and risks.

Once too many programs are approved to start, the budgeting process exacerbates problems. Because programs are funded annually and department wide, cross-portfolio priorities have not been established, competition for funding continues over time, forcing programs to view success as the ability to secure the next funding increment rather than delivering capabilities when and as promised. As a result, there is pressure to suppress bad news about programs, which could endanger funding and support, as well as to skip testing because of its high cost. Concurrently, when faced with budget constraints, senior officials tend to make across- the-board cuts to all programs rather than make the hard decisions as to which ones to keep and which ones to cancel or cut back. In many cases, the system delivers less performance than promised when initial investment decisions were made.

So, the condition we encounter time after time describes a predictable outcome. The acquisition environment encourages launching product developments that embody more technical unknowns and less knowledge about the performance and production risks they entail. A new weapon system is encouraged to possess performance features that significantly distinguish it from other systems and promises the best capability. A new program will not be approved unless its costs fall within forecasts of available funds and, therefore, looks affordable. Because cost and schedule estimates are comparatively soft at the time, successfully competing for funds encourages the program’s estimates to be squeezed into the funds available. Consequently, DOD program managers have incentives to promote performance features and design characteristics that rely on immature technologies and decision makers lack the knowledge they need to make good decisions.

The Path to Better Decisions

A path can be laid out to make decisions that will lead to better program choices and better outcomes. Much of this is known and has been recommended by one study or another. GAO itself has issued hundreds of reports. The key recommendations we have made have been focused on the product development process: constraining individual program requirements by working within available resources and by leveraging systems engineering; establishing clear business cases for each individual investment; enabling science and technology organizations to shoulder the ensuring that the workforce is capable of managing requirements trades, source selection, and knowledge-based acquisition strategies; and establishing and enforcing controls to ensure that appropriate knowledge is captured and used at critical junctures before moving programs forward and investing more money.

As I have outlined above, however, setting the right conditions for successful acquisitions outcomes goes beyond product development. We are currently examining how to bring discipline to the Department’s requirements and budgetary process and the role played by the program manager.

As we conduct this work, we will be asking who is currently accountable for acquisition decisions; who should be held accountable; how much deviation from the original business case is allowed before the entire program investment is reconsidered; and what is the penalty when investments do not result in meeting promised warfighter needs?

We can make hard, but thoughtful, decisions now or postpone them, allowing budgetary realities to force draconian decisions later.

Mr. Chairman, this concludes my prepared statement. I would be happy to respond to any questions that you or other members of the subcommittee may have.

Contacts and Staff Acknowledgments

For further information regarding this testimony, please contact Katherine V. Schinasi at (202) 512-4841 or schinasik@gao.gov. Individuals making key contributions to this testimony included Paul L. Francis, David B. Best, David J. Hand, Alan R. Frazier, Adam Vodraska, and Lily J. Chin.

Related GAO Products

Space Acquisitions: Stronger Development Practices and Investment Planning Needed to Address Continuing Problems. GAO-05-891T. Washington, D.C.: July 12, 2005.

Air Force Procurement: Protests Challenging Role of Biased Official Sustained. GAO-05-436T. Washington, D.C.: April 14, 2005.

Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications for Tactical Aircraft Modernization. GAO-05-591T. Washington, D.C.: April 6, 2005.

Defense Acquisitions: Assessments of Selected Major Weapon Programs. GAO-05-301. Washington, D.C.: March 31, 2005.

Defense Acquisitions: Future Combat Systems Challenges and Prospects for Success. GAO-05-428T. Washington, D.C.: March 16, 2005.

Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004.

Defense Acquisitions: DOD’s Revised Policy Emphasizes Best Practices, but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10, 2003.

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Program Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.