Defense Acquisitions: Charting a Course for Lasting Reform
Released: 2009-04-30T00:00:00

Letter

The Case for Reform

There can be little doubt that we can—and must—get better outcomes from our weapon system investments. As can be seen in table 1 below, cost growth and schedule delays in DOD’s portfolio of weapon systems have been significant.

Since 2003, DOD’s portfolio of major defense acquisition programs has grown from 77 to 96 programs and its investment in those programs has grown from $1.2 trillion to $1.6 trillion (fiscal year 2009 dollars). The total cost growth for DOD’s portfolio of major defense acquisition programs is higher than it was 5 years ago, but at $296 billion, it is actually less than the 2007 portfolio’s cost growth of $301 billion. For DOD’s 2008 portfolio of programs, total research and development costs are 42 percent higher than originally estimated, and the average delay in delivering initial capabilities is 22 months. In addition, 42 percent of the programs reported a 25 percent or more increase in acquisition unit costs. DOD’s performance in some of these areas is driven by older, underperforming programs as newer programs, on average, have not yet shown the same degree of cost and schedule growth. Of the programs in the 2008 portfolio that reported comparable cost data, 75 percent (69 programs) reported increases in research and development costs since their first estimate, and 69 percent (64 programs) reported increases in total acquisition costs. Quantities have been reduced by 25 percent or more for 15 of the programs in the 2008 portfolio.

The overall performance of this portfolio is one indicator of how well DOD’s acquisition system generates the return on investment it promises to the warfighter, Congress, and the taxpayer. Another is the effect cost increases have on DOD’s buying power for individual systems, as demonstrated by changes in program acquisition unit costs. Some examples that illustrate the effect of lost buying power are shown in table 2 below.

There is no single measure that perfectly explains every variable behind cost and schedule changes in weapon systems. For example, the total cost of a weapon system can increase because more quantities are added, without necessarily being indicative of a problem. On the other hand, the total cost can stay the same while quantities are significantly reduced—a clear indication of a problem. While there can be legitimate debate over which set of measures are the best explanation of the problem, as table 1 shows, there can be no debate over the fact that the problem is significant and calls for action.

What Needs to Change?

DOD’s key processes for setting requirements, providing funding, and managing acquisition programs have institutionalized some underlying causes for persistent problems in weapon system programs. As illustrated in figure 1 below, collectively, these processes create pressures to promise high performance, keep cost estimates low, and proceed with calendar- driven versus knowledge-driven schedules.

DOD’s processes for identifying warfighter needs, funding programs, and developing and procuring weapon systems—which collectively define DOD’s overall weapon system investment strategy—do not work together to provide the best value to the warfighter and to the taxpayer. Instead, DOD largely continues to define warfighting needs and make investment decisions on a service-by-service and individual platform basis. As a result, DOD does not effectively address joint warfighting needs and commits to more programs than it has resources for, thus creating unhealthy competition for funding. At the individual program level, a military service typically establishes and DOD approves a business case containing requirements that are not fully understood and cost and schedule estimates that are based on optimistic assumptions rather than on sufficient knowledge. This makes it impossible to successfully execute the program within established cost, schedule, and performance targets. Because DOD’s requirements, funding, and acquisition processes are led by different organizations, it is difficult to hold any one person or organization accountable for saying no to a proposed program or for ensuring that the department’s portfolio of programs is balanced. Frequent turnover in leadership positions in the department exacerbates the problem. As of March 2009, the average tenure of the Under Secretary of Defense for Acquisition, Technology and Logistics over the past 22 years has been only about 20 months.

Requirements Process

DOD’s requirements determination process—the Joint Capabilities and Integration Development System (JCIDS)—provides a framework for reviewing and validating needs. However, it does not adequately prioritize those needs from a joint, departmentwide perspective and lacks the agility to meet changing warfighter demands. We recently reviewed JCIDS documentation related to new capability proposals and found that most were sponsored by the military services with little involvement from the joint community, including the combatant commands. By continuing to primarily rely on stovepiped solutions to address capability needs, DOD may be losing opportunities to improve joint warfighting capabilities and reduce the duplication of capabilities in some areas. Furthermore, the vast majority of capability proposals that enter the JCIDS process are validated or approved without accounting for the resources or technologies that will be needed to acquire the desired capabilities. As a result, the process produces more demand for new weapon system programs than available resources can support.

Funding Process

The funding of proposed programs takes place through a separate process—the Planning, Programming, Budgeting, and Execution system, which is not synchronized with JCIDS. We recently reviewed the impact of the funding process on major defense acquisition programs and found that the process does not produce an accurate picture of DOD’s resource needs for weapon system programs. The estimated cost of many of the programs we reviewed exceeded the funding levels planned for and reflected in the Future Years Defense Program—DOD’s long-term investment strategy. Rather than limit the number and size of programs or adjust requirements, the funding process attempts to accommodate programs. This creates an unhealthy competition for funds that encourages sponsors of weapon system programs to pursue over ambitious capabilities and to underestimate costs. With too many programs underway for the available resources and high cost growth occurring in many programs, DOD must make up for funding shortfalls by shifting funds from one program to pay for another, reducing system or in capabilities, cutting procurement quantities, stretching out programs, rare cases terminating programs. Such actions not only create instability inDOD’s weapon system portfolio, they further obscure the true future cost s of current commitments, making it difficult to ma ke informed investment decisions.

Acquisition Process

At the program level, the key cause of poor outcomes is the approval of programs with business cases that contain inadequate knowledge a bout requirements and the resources—funding, time, technologies, and people—needed to execute them. In a sense, the business cases are compromised to reconcile the disparate pressures imposed by the requirements and funding processes. We analyze the soundness of individual business cases at various points in the acquisition proce through the lens of three knowledge points that are indicators of technology, design, and production maturity. These knowledge points are ss consistent with best practices for product development. Some key observations on each follow.

Knowledge point 1: Resources and requirements match. Achieving a high level of technology maturity by the start of system development is an important indicator of whether this match has been made. This means that the technologies needed to meet essential product requirements have been demonstrated to work in their intended environment. In addition, the developer has completed a preliminary design of the product that shows the design is feasible. DOD’s acquisition policy and statute both require that technologies should be demonstrated in a relevant environment prior to starting development—a slightly lower standard than the best practice. Since 2003, there has been a significant increase in the technology maturity of DOD programs at the start of system development; however, few programs have met the best practices standard. In our most recent assessment, only 4 of the 36 programs that provided data on technology maturity at development start did so with fully mature critical technologies. In addition, only 4 of the 36 programs that held a preliminary design review did so before development start; the remaining programs held the review, on average, 31 months after development began.

Knowledge point 2: Product design is stable. This point occurs when a program determines that a product’s design will meet customer requirements, as well as cost, schedule, and reliability targets. A best practice is to achieve design stability at the system-level critical design review, usually held midway through system development. Completion of at least 90 percent of engineering drawings at this point provides tangible evidence that the product’s design is stable, and a prototype demonstration shows that the design is capable of meeting performance requirements. Of the 29 programs in our most recent assessment that have held a system-level critical design review, 7 reported having a stable design. However, the level of design knowledge attained by the critical design review has been increasing over time.

Knowledge point 3: Manufacturing processes are mature. This point is achieved when it has been demonstrated that the developer can manufacture the product within cost, schedule, and quality targets. A best practice is to ensure that all critical manufacturing processes are in statistical control—that is, they are repeatable, sustainable, and capable of consistently producing parts within the product’s quality tolerances and standards—at the start of production. Identifying key product characteristics and the associated critical manufacturing processes is a key initial step to ensuring production elements are stable and in control. In our most recent assessment, only 4 of the 23 programs that had made a production decision identified key product characteristics or associated critical manufacturing processes. However, it should also be noted that 4 of the 17 programs that are scheduled to make a production decision in the next 3 years have already identified key product characteristics or associated critical manufacturing processes.

When programs do not follow a knowledge-based approach to acquisition, high levels of uncertainty about requirements, technologies, and design often exist at the start of a development program. As a result, cost estimates and related funding needs are often understated. Our analysis of service and independent cost estimates for 20 major weapon system programs shows that while the independent estimates were somewhat higher, both estimates were too low in most cases. The programs we reviewed frequently lacked sufficient knowledge and detail about planned program content for developing sound cost estimates. Without this knowledge, cost estimators must rely heavily on parametric analysis and assumptions. A cost estimate is then usually presented to decision makers as a single, or point, estimate that is expected to represent the most likely cost of the program but provides no information about the range of risk and uncertainty or level of confidence associated with the estimate. Second, the basic principles of systems engineering are not being followed when preliminary and critical design reviews are not conducted on time or with insufficient information. Further, testing a fully integrated, capable, production-representative prototype is essential to confirm the maturity of the design and to minimize cost growth in production. Yet of the 33 programs in our most recent assessment that reported they plan to test such prototypes, only 17 planned to do so before the production decision.

Recent and Proposed Reform Efforts Could Improve Weapon Programs

There is widespread recognition of the problems that affect the acquisition system and DOD and the Congress have taken and proposed several steps to remedy them. Changes have been introduced to improve the department’s processes for determining warfighter needs and funding programs, establish sound business cases for starting acquisition programs, and execute programs more effectively.

Efforts to Prioritize Needs and Manage Resources

DOD has recently implemented measures to better address the needs of the joint warfighter and align the demand for weapon systems with available resources. The Joint Requirements and Oversight Council, for example, has been doing more to seek out and consider input from the combatant commands (COCOMs)—the principal joint warfighting customer in DOD—through regular trips and meetings to discuss capability needs and resource issues. This may help alleviate concerns that the COCOMs have raised in the past that their needs have not been adequately addressed through the department’s requirements process. In addition, DOD has taken action over the past few years to field capabilities that are urgently needed for Iraq and Afghanistan, such as Unmanned Aerial Systems and Mine Resistant Ambush Protected vehicles. While these efforts have helped DOD meet the needs of the warfighter, Secretary Gates noted earlier this year that the department must figure out how to institutionalize the acquisition of urgently-needed capabilities rather than having to do so, on an ad hoc basis. Having greater combatant command involvement in determining requirements, as emphasized in the House and Senate Armed Services Committees’ proposed acquisition reform legislation, would help to ensure that near-term needs are addressed. We have recommended additional actions that DOD should take to balance the needs of the military services, COCOMs, and other defense components, including establishing an analytic approach to determine the relative importance of capabilities and providing the COCOMs with additional resources to establish robust analytical capabilities for identifying and assessing their warfighting needs.

DOD has also recently established a capability portfolio management framework to facilitate more strategic choices for allocating resources through the funding process. Capability portfolios have been set up to advise the department on how to optimize investments within individual capability areas, but portfolio managers do no have independent decision- making authority over determining requirements and funding. Although it is too soon to assess the impact of the portfolios, according to some DOD officials, portfolio managers have provided key input and recommendations during the budget process this year. However, while these portfolio managers may improve the management of individual capability areas, there still needs to be higher level DOD attention to improving the match between the number of major defense acquisition programs and available funding. The Secretary of Defense recently recommended the termination of several major weapon programs, which will help bring the portfolio into balance better and prioritize capability needs. Sustaining a balance over the long term will require DOD to improve the way it makes decisions about which programs to pursue or not pursue. Legislative proposals to promote greater consideration of trade-offs in the cost, schedule, and performance of individual programs and the development of better mechanisms to ensure this happens before a program begins could help make such decisions part of a disciplined process versus the product of extraordinary action.

Efforts to Establish Sound Programs

Recognizing the need for more discipline and accountability in the acquisition process, Congress and DOD have recently introduced several initiatives that could provide a foundation for establishing sound, knowledge-based business cases for individual weapon programs. These initiatives require programs to invest more time and resources in the front- end of the acquisition process—refining weapon system concepts through early systems engineering, developing technologies, and building prototypes—before starting system development. In the past, weapon programs often rushed into systems development before they were ready, in part because the department’s acquisition process did not require early formal milestone reviews and programs would rarely be terminated once underway. If implemented, these changes could help programs replace risk with knowledge, thereby increasing the chances of developing weapon systems within cost and schedule targets while meeting user needs. However, DOD must ensure that these changes to the acquisition process are consistently implemented and reflected in decisions on individual programs. Several of the provisions in the proposed House and Senate legislation will codify DOD policies that are not yet being implemented consistently in weapon programs. Some of the key changes being introduced by DOD and the Congress to establish knowledge-based business cases for weapon programs include: Increased emphasis on early systems engineering activities and the enhancement of systems engineering capabilities within the department; A requirement for competitive prototyping of a proposed weapon system or key system elements during the technology development phase; Certification that critical technologies have been demonstrated in a relevant environment before the start of system development, employing independent technology readiness assessments to make these determinations; Early milestone reviews for programs going through the pre–systems Conducting preliminary design reviews before starting system Requiring early cost estimates for the milestone decision to move into technology development phase; and Elevating the role of independent cost estimates in the acquisition process, by establishing a director or principal advisor for cost estimating who reports to the Secretary of Defense and the Congress.

Efforts to Improve Program Execution

There have also been several policy and legislative provisions introduced to improve the execution of weapon system programs. For example, to address the problem of requirements creep that has often plagued programs in the past, review boards are being established to identify and mitigate technical risks and evaluate the impact of any potential requirements changes on ongoing programs. In addition, to improve accountability in managing programs, DOD has established a policy instituting formal agreements among program managers, their acquisition executives, and the user community setting forth common program goals and the resources that will be provided to reach these goals. Further, DOD acquisition policy now incorporates a requirement that program managers sign tenure agreements so that their tenure will correspond to the next major acquisition milestone review closest to 4 years. The House and Senate reform legislation introduces a number of steps to monitor and oversee the progress of existing programs that started before the recent certification requirements were put in place for gaining approval to enter system development, and programs where DOD waived the certification requirements for one reason or another. The House proposal also includes establishing a principal advisor for performance assessment—a position that will focus primarily on program execution. In addition, requiring any program that the department determines must be continued following a Nunn-McCurdy breach, to go back through a major milestone review, should help ensure that programs with significant cost problems are not allowed to continue without a sound business case.

Concluding Observations on Achieving Lasting Reform

I would like to offer a few thoughts about other factors that should be considered so that we make the most out of today’s opportunity for meaningful change. First, I think it is useful to think of the processes that affect weapon system outcomes (requirements, funding, and acquisition) as being in a state of equilibrium. Poor outcomes—delays, cost growth, and reduced quantities—have been persistent for decades. If we think of these processes as merely “broken”, then some targeted repairs should fix them. I think the challenge is greater than that. If we think of these processes as being in equilibrium, where their inefficiencies are implicitly accepted as the cost of doing business, then the challenge for getting better outcomes is greater. Seen in this light, it will take considerable and sustained effort to change the incentives and inertia that reinforce the status quo.

Second, while actions taken and proposed by DOD and Congress are constructive and will serve to improve acquisition outcomes, one has to ask the question why extraordinary actions are needed to force practices that should occur normally. The answer to this question will shed light on the cultural or environmental forces that operate against sound management practices. For reforms to work, they will have to address these forces as well. For example, there are a number of proposals to make cost estimates more rigorous and realistic, but do these address all of the reasons why estimates are not already realistic? Clearly, more independence, methodological rigor, and better information about risk areas like technology will make estimates more realistic. On the other hand, realism is compromised as the competition for funding encourages programs to appear affordable. Also, when program sponsors present a program as more than a weapon system, but rather as essential to new fighting concepts, pressures exist to accept less than rigorous cost estimates. Reform must recognize and counteract these pressures as well.

Third, decisions on individual systems must reinforce good practices. Programs that have pursued risky and unexecutable acquisition strategies have succeeded in winning approval and funding. If reform is to succeed, then programs that present realistic strategies and resource estimates must succeed in winning approval and funding. Those programs that continue past practices of pushing unexecutable strategies must be denied funding before they begin. This will take the cooperative efforts of DOD and Congress.

Fourth, consideration should be given to setting some limits on what is a reasonable length of time for developing a system. For example, if a program has to complete development within 5 or 6 years, this could serve as a basis to constrain requirements and exotic programs. It would also serve to get capability in the hands of the warfighter sooner.

Fifth, the institutional resources we have must match the outcomes we desire. For example, if more work must be done to reduce technical risk before development start—milestone B—DOD needs to have the organizational, people, and financial resources to do so. Once a program is approved for development, program offices and testing organizations must have the workforce with the requisite skills to manage and oversee the effort. Contracting instruments must be used that match the needs of the acquisition and protect the government’s interests. Finally, DOD must be judicious and consistent in how it relies on contractors.

Mr. Chairman, this concludes my prepared statement. I would be happy to answer any questions you may have at this time.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.