The Case for Reform

There can be little doubt that we can—and must—get better outcomes  from our weapon system investments. As can be seen in table 1 below,  cost growth and schedule delays in DOD’s portfolio of weapon systems  have been significant.
Since 2003, DOD’s portfolio of major defense acquisition programs has  grown from 77 to 96 programs and its investment in those programs has  grown from $1.2 trillion to $1.6 trillion (fiscal year 2009 dollars). The total  cost growth for DOD’s portfolio of major defense acquisition programs is  higher than it was 5 years ago, but at $296 billion, it is actually less than  the 2007 portfolio’s cost growth of $301 billion. For DOD’s 2008 portfolio  of programs, total research and development costs are 42 percent higher  than originally estimated, and the average delay in delivering initial  capabilities is 22 months. In addition, 42 percent of the programs reported  a 25 percent or more increase in acquisition unit costs. DOD’s  performance in some of these areas is driven by older, underperforming  programs as newer programs, on average, have not yet shown the same  degree of cost and schedule growth. Of the programs in the 2008 portfolio  that reported comparable cost data, 75 percent (69 programs) reported  increases in research and development costs since their first estimate, and  69 percent (64 programs) reported increases in total acquisition costs.  Quantities have been reduced by 25 percent or more for 15 of the  programs in the 2008 portfolio.
The overall performance of this portfolio is one indicator of how well  DOD’s acquisition system generates the return on investment it promises  to the warfighter, Congress, and the taxpayer. Another is the effect cost  increases have on DOD’s buying power for individual systems, as  demonstrated by changes in program acquisition unit costs. Some  examples that illustrate the effect of lost buying power are shown in table  2 below.
There is no single measure that perfectly explains every variable behind  cost and schedule changes in weapon systems. For example, the total cost  of a weapon system can increase because more quantities are added,  without necessarily being indicative of a problem. On the other hand, the  total cost can stay the same while quantities are significantly reduced—a  clear indication of a problem. While there can be legitimate debate over  which set of measures are the best explanation of the problem, as table 1  shows, there can be no debate over the fact that the problem is significant  and calls for action.

	What Needs to Change?

DOD’s key processes for setting requirements, providing funding, and  managing acquisition programs have institutionalized some underlying  causes for persistent problems in weapon system programs. As illustrated  in figure 1 below, collectively, these processes create pressures to promise  high performance, keep cost estimates low, and proceed with calendar- driven versus knowledge-driven schedules.
DOD’s processes for identifying warfighter needs, funding programs, and  developing and procuring weapon systems—which collectively define  DOD’s overall weapon system investment strategy—do not work together  to provide the best value to the warfighter and to the taxpayer. Instead,  DOD largely continues to define warfighting needs and make investment  decisions on a service-by-service and individual platform basis. As a result,  DOD does not effectively address joint warfighting needs and commits to  more programs than it has resources for, thus creating unhealthy  competition for funding. At the individual program level, a military service  typically establishes and DOD approves a business case containing  requirements that are not fully understood and cost and schedule  estimates that are based on optimistic assumptions rather than on  sufficient knowledge. This makes it impossible to successfully execute the  program within established cost, schedule, and performance targets.  Because DOD’s requirements, funding, and acquisition processes are led  by different organizations, it is difficult to hold any one person or  organization accountable for saying no to a proposed program or for  ensuring that the department’s portfolio of programs is balanced. Frequent  turnover in leadership positions in the department exacerbates the  problem. As of March 2009, the average tenure of the Under Secretary of  Defense for Acquisition, Technology and Logistics over the past 22 years  has been only about 20 months.

		Requirements Process

DOD’s requirements determination process—the Joint Capabilities and  Integration Development System (JCIDS)—provides a framework for  reviewing and validating needs. However, it does not adequately prioritize  those needs from a joint, departmentwide perspective and lacks the agility  to meet changing warfighter demands. We recently reviewed JCIDS  documentation related to new capability proposals and found that most  were sponsored by the military services with little involvement from the  joint community, including the combatant commands. By continuing to  primarily rely on stovepiped solutions to address capability needs, DOD  may be losing opportunities to improve joint warfighting capabilities and  reduce the duplication of capabilities in some areas. Furthermore, the vast  majority of capability proposals that enter the JCIDS process are validated  or approved without accounting for the resources or technologies that will  be needed to acquire the desired capabilities. As a result, the process  produces more demand for new weapon system programs than available  resources can support.

		Funding Process

The funding of proposed programs takes place through a separate  process—the Planning, Programming, Budgeting, and Execution system,  which is not synchronized with JCIDS. We recently reviewed the impact of  the funding process on major defense acquisition programs and found that  the process does not produce an accurate picture of DOD’s resource needs  for weapon system programs. The estimated cost of many of the  programs we reviewed exceeded the funding levels planned for and  reflected in the Future Years Defense Program—DOD’s long-term  investment strategy. Rather than limit the number and size of programs or  adjust requirements, the funding process attempts to accommodate  programs. This creates an unhealthy competition for funds that  encourages sponsors of weapon system programs to pursue over ambitious capabilities and to underestimate costs. With too many  programs underway for the available resources and high cost growth  occurring in many programs, DOD must make up for funding shortfalls by  shifting funds from one program to pay for another, reducing system  or in  capabilities, cutting procurement quantities, stretching out programs,  rare cases terminating programs. Such actions not only create instability inDOD’s weapon system portfolio, they further obscure the true future cost s  of current commitments, making it difficult to ma ke informed investment  decisions.

		Acquisition Process

At the program level, the key cause of poor outcomes is the approval of  programs with business cases that contain inadequate knowledge a bout  requirements and the resources—funding, time, technologies, and  people—needed to execute them. In a sense, the business cases are compromised to reconcile the disparate pressures imposed by the  requirements and funding processes. We analyze the soundness of  individual business cases at various points in the acquisition proce through the lens of three knowledge points that are indicators of  technology, design, and production maturity. These knowledge points are  ss  consistent with best practices for product development. Some key  observations on each follow.
Knowledge point 1: Resources and requirements match. Achieving a  high level of technology maturity by the start of system development is an  important indicator of whether this match has been made. This means  that the technologies needed to meet essential product requirements have  been demonstrated to work in their intended environment. In addition, the  developer has completed a preliminary design of the product that shows  the design is feasible. DOD’s acquisition policy and statute both require  that technologies should be demonstrated in a relevant environment prior  to starting development—a slightly lower standard than the best practice.  Since 2003, there has been a significant increase in the technology  maturity of DOD programs at the start of system development; however,  few programs have met the best practices standard. In our most recent  assessment, only 4 of the 36 programs that provided data on technology  maturity at development start did so with fully mature critical  technologies. In addition, only 4 of the 36 programs that held a  preliminary design review did so before development start; the remaining  programs held the review, on average, 31 months after development  began.
Knowledge point 2: Product design is stable. This point occurs when a  program determines that a product’s design will meet customer  requirements, as well as cost, schedule, and reliability targets. A best  practice is to achieve design stability at the system-level critical design  review, usually held midway through system development. Completion of  at least 90 percent of engineering drawings at this point provides tangible  evidence that the product’s design is stable, and a prototype  demonstration shows that the design is capable of meeting performance  requirements. Of the 29 programs in our most recent assessment that have  held a system-level critical design review, 7 reported having a stable  design. However, the level of design knowledge attained by the critical  design review has been increasing over time.
Knowledge point 3: Manufacturing processes are mature. This point  is achieved when it has been demonstrated that the developer can  manufacture the product within cost, schedule, and quality targets. A best  practice is to ensure that all critical manufacturing processes are in  statistical control—that is, they are repeatable, sustainable, and capable of  consistently producing parts within the product’s quality tolerances and  standards—at the start of production. Identifying key product  characteristics and the associated critical manufacturing processes is a  key initial step to ensuring production elements are stable and in control.  In our most recent assessment, only 4 of the 23 programs that had made a  production decision identified key product characteristics or associated  critical manufacturing processes. However, it should also be noted that 4  of the 17 programs that are scheduled to make a production decision in  the next 3 years have already identified key product characteristics or  associated critical manufacturing processes.
When programs do not follow a knowledge-based approach to acquisition,  high levels of uncertainty about requirements, technologies, and design  often exist at the start of a development program. As a result, cost  estimates and related funding needs are often understated. Our analysis of  service and independent cost estimates for 20 major weapon system  programs shows that while the independent estimates were somewhat  higher, both estimates were too low in most cases. The programs we  reviewed frequently lacked sufficient knowledge and detail about planned  program content for developing sound cost estimates. Without this  knowledge, cost estimators must rely heavily on parametric analysis and  assumptions. A cost estimate is then usually presented to decision makers  as a single, or point, estimate that is expected to represent the most likely  cost of the program but provides no information about the range of risk  and uncertainty or level of confidence associated with the estimate.  Second, the basic principles of systems engineering are not being followed  when preliminary and critical design reviews are not conducted on time or  with insufficient information. Further, testing a fully integrated, capable,  production-representative prototype is essential to confirm the maturity of  the design and to minimize cost growth in production. Yet of the 33  programs in our most recent assessment that reported they plan to test  such prototypes, only 17 planned to do so before the production decision.

	Recent and Proposed Reform Efforts Could Improve Weapon Programs

There is widespread recognition of the problems that affect the acquisition  system and DOD and the Congress have taken and proposed several steps  to remedy them. Changes have been introduced to improve the  department’s processes for determining warfighter needs and funding  programs, establish sound business cases for starting acquisition  programs, and execute programs more effectively.

		Efforts to Prioritize Needs and Manage Resources

DOD has recently implemented measures to better address the needs of  the joint warfighter and align the demand for weapon systems with  available resources. The Joint Requirements and Oversight Council, for  example, has been doing more to seek out and consider input from the  combatant commands (COCOMs)—the principal joint warfighting  customer in DOD—through regular trips and meetings to discuss  capability needs and resource issues. This may help alleviate concerns that  the COCOMs have raised in the past that their needs have not been  adequately addressed through the department’s requirements process. In  addition, DOD has taken action over the past few years to field capabilities  that are urgently needed for Iraq and Afghanistan, such as Unmanned  Aerial Systems and Mine Resistant Ambush Protected vehicles. While  these efforts have helped DOD meet the needs of the warfighter, Secretary  Gates noted earlier this year that the department must figure out how to  institutionalize the acquisition of urgently-needed capabilities rather than  having to do so, on an ad hoc basis. Having greater combatant command  involvement in determining requirements, as emphasized in the House and  Senate Armed Services Committees’ proposed acquisition reform  legislation, would help to ensure that near-term needs are addressed. We  have recommended additional actions that DOD should take to balance  the needs of the military services, COCOMs, and other defense  components, including establishing an analytic approach to determine the  relative importance of capabilities and providing the COCOMs with  additional resources to establish robust analytical capabilities for  identifying and assessing their warfighting needs.
DOD has also recently established a capability portfolio management  framework to facilitate more strategic choices for allocating resources  through the funding process. Capability portfolios have been set up to  advise the department on how to optimize investments within individual  capability areas, but portfolio managers do no have independent decision- making authority over determining requirements and funding. Although it  is too soon to assess the impact of the portfolios, according to some DOD  officials, portfolio managers have provided key input and  recommendations during the budget process this year. However, while  these portfolio managers may improve the management of individual  capability areas, there still needs to be higher level DOD attention to  improving the match between the number of major defense acquisition  programs and available funding. The Secretary of Defense recently  recommended the termination of several major weapon programs, which  will help bring the portfolio into balance better and prioritize capability  needs. Sustaining a balance over the long term will require DOD to  improve the way it makes decisions about which programs to pursue or  not pursue. Legislative proposals to promote greater consideration of  trade-offs in the cost, schedule, and performance of individual programs  and the development of better mechanisms to ensure this happens before  a program begins could help make such decisions part of a disciplined  process versus the product of extraordinary action.

		Efforts to Establish Sound Programs

Recognizing the need for more discipline and accountability in the  acquisition process, Congress and DOD have recently introduced several  initiatives that could provide a foundation for establishing sound,  knowledge-based business cases for individual weapon programs. These  initiatives require programs to invest more time and resources in the front- end of the acquisition process—refining weapon system concepts through  early systems engineering, developing technologies, and building  prototypes—before starting system development. In the past, weapon  programs often rushed into systems development before they were ready,  in part because the department’s acquisition process did not require early  formal milestone reviews and programs would rarely be terminated once  underway. If implemented, these changes could help programs replace  risk with knowledge, thereby increasing the chances of developing  weapon systems within cost and schedule targets while meeting user  needs. However, DOD must ensure that these changes to the acquisition  process are consistently implemented and reflected in decisions on  individual programs. Several of the provisions in the proposed House and  Senate legislation will codify DOD policies that are not yet being  implemented consistently in weapon programs. Some of the key changes  being introduced by DOD and the Congress to establish knowledge-based  business cases for weapon programs include:  Increased emphasis on early systems engineering activities and the  enhancement of systems engineering capabilities within the department;    A requirement for competitive prototyping of a proposed weapon system  or key system elements during the technology development phase;    Certification that critical technologies have been demonstrated in a  relevant environment before the start of system development, employing  independent technology readiness assessments to make these  determinations;    Early milestone reviews for programs going through the pre–systems    Conducting preliminary design reviews before starting system    Requiring early cost estimates for the milestone decision to move into  technology development phase; and    Elevating the role of independent cost estimates in the acquisition process,  by establishing a director or principal advisor for cost estimating who  reports to the Secretary of Defense and the Congress.

		Efforts to Improve Program Execution

There have also been several policy and legislative provisions introduced  to improve the execution of weapon system programs. For example, to  address the problem of requirements creep that has often plagued  programs in the past, review boards are being established to identify and  mitigate technical risks and evaluate the impact of any potential  requirements changes on ongoing programs. In addition, to improve  accountability in managing programs, DOD has established a policy  instituting formal agreements among program managers, their acquisition  executives, and the user community setting forth common program goals  and the resources that will be provided to reach these goals. Further, DOD  acquisition policy now incorporates a requirement that program managers  sign tenure agreements so that their tenure will correspond to the next  major acquisition milestone review closest to 4 years. The House and  Senate reform legislation introduces a number of steps to monitor and  oversee the progress of existing programs that started before the recent  certification requirements were put in place for gaining approval to enter  system development, and programs where DOD waived the certification  requirements for one reason or another. The House proposal also includes  establishing a principal advisor for performance assessment—a position  that will focus primarily on program execution. In addition, requiring any  program that the department determines must be continued following a  Nunn-McCurdy breach, to go back through a major milestone review,  should help ensure that programs with significant cost problems are not  allowed to continue without a sound business case.

	Concluding Observations on Achieving Lasting Reform

I would like to offer a few thoughts about other factors that should be  considered so that we make the most out of today’s opportunity for  meaningful change. First, I think it is useful to think of the processes that  affect weapon system outcomes (requirements, funding, and acquisition)  as being in a state of equilibrium. Poor outcomes—delays, cost growth,  and reduced quantities—have been persistent for decades. If we think of  these processes as merely “broken”, then some targeted repairs should fix  them. I think the challenge is greater than that. If we think of these  processes as being in equilibrium, where their inefficiencies are implicitly  accepted as the cost of doing business, then the challenge for getting  better outcomes is greater. Seen in this light, it will take considerable and  sustained effort to change the incentives and inertia that reinforce the  status quo.
Second, while actions taken and proposed by DOD and Congress are  constructive and will serve to improve acquisition outcomes, one has to  ask the question why extraordinary actions are needed to force practices  that should occur normally. The answer to this question will shed light on  the cultural or environmental forces that operate against sound  management practices. For reforms to work, they will have to address  these forces as well. For example, there are a number of proposals to  make cost estimates more rigorous and realistic, but do these address all  of the reasons why estimates are not already realistic? Clearly, more  independence, methodological rigor, and better information about risk  areas like technology will make estimates more realistic. On the other  hand, realism is compromised as the competition for funding encourages  programs to appear affordable. Also, when program sponsors present a  program as more than a weapon system, but rather as essential to new  fighting concepts, pressures exist to accept less than rigorous cost  estimates. Reform must recognize and counteract these pressures as well.
Third, decisions on individual systems must reinforce good practices.  Programs that have pursued risky and unexecutable acquisition strategies  have succeeded in winning approval and funding. If reform is to succeed,  then programs that present realistic strategies and resource estimates  must succeed in winning approval and funding. Those programs that  continue past practices of pushing unexecutable strategies must be denied  funding before they begin. This will take the cooperative efforts of DOD  and Congress.
Fourth, consideration should be given to setting some limits on what is a  reasonable length of time for developing a system. For example, if a  program has to complete development within 5 or 6 years, this could serve  as a basis to constrain requirements and exotic programs. It would also  serve to get capability in the hands of the warfighter sooner.
Fifth, the institutional resources we have must match the outcomes we  desire. For example, if more work must be done to reduce technical risk  before development start—milestone B—DOD needs to have the  organizational, people, and financial resources to do so. Once a program is  approved for development, program offices and testing organizations must  have the workforce with the requisite skills to manage and oversee the  effort. Contracting instruments must be used that match the needs of the  acquisition and protect the government’s interests. Finally, DOD must be  judicious and consistent in how it relies on contractors.
Mr. Chairman, this concludes my prepared statement. I would be happy to  answer any questions you may have at this time.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.

Summary:
Why GAO Did This Study

Since 1990, GAO has designated the Department of Defense's (DOD) management of its major weapon acquisitions as a high-risk area; however DOD's problems delivering weapon systems on time, at the estimated cost, in the planned amounts, and with the promised performance go back decades. Congress and DOD have continually explored ways to improve acquisition outcomes, yet problems persist. The committee asked GAO to testify on measures needed to reform the acquisition of major weapon systems and related legislative proposals. Specifically, this statement will describe the poor outcomes on weapon system investments that make reform imperative; attributes of the requirements, funding, and acquisition processes that will need to change for reform to be effective; and positive steps that Congress and DOD have taken to improve weapon program outcomes. The statement will also examine other factors that should be considered as the committee moves forward with its reform efforts. The testimony is drawn from GAO's body of work on DOD's requirements, funding, and acquisition processes. GAO has made numerous recommendations aimed at improving DOD's management of its major weapon acquisitions, but it is not making any new recommendations in this testimony.

What GAO Found

DOD must get a better return on investment from its weapon system programs. Since fiscal year 2003, DOD has increased the number of major defense acquisition programs and its overall investment in them. The cumulative cost growth for DOD's programs is higher than it was 5 years ago, but at $296 billion (fiscal year 2009 dollars), it is less than last year when adjusted for inflation. For DOD's 2008 portfolio of programs, research and development costs are now 42 percent higher than originally estimated and the average delay in delivering initial capabilities has increased to 22 months. These problems have roots in not only the acquisition process, but also in the requirements and funding processes. Collectively, these processes create pressures to demand high performance, keep cost estimates low, and proceed with calendar-driven versus knowledge-driven schedules. These processes also do not adequately prioritize needs from a joint, departmentwide perspective, respond to changing warfighter demands, or constrain the number of programs to a level that is supportable by available resources. Programs are allowed to enter and proceed through the acquisition process with requirements that are not fully understood, cost and schedule estimates that are based on optimistic assumptions, and a lack of sufficient knowledge about technology, design, and manufacturing. Congressionally-mandated and DOD-initiated changes to the acquisition system could provide the basis for sounder programs and improved acquisition outcomes. The committee's proposed legislation dealing with requirements, systems engineering, technology and integration risk assessment, and cost estimation--also address areas in need of reform. However, past reform efforts have failed to produce lasting change. To make the most out of this opportunity, the weapons acquisition environment and the incentives inherent within it will also have to be confronted and addressed.