Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes
Published: 2001-03-08T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: This report examines how best practices offer improvements to the way the Department of Defense defines and matches weapon system requirements to available resources such as cost, schedule, and mature technologies. GAO identified three factors that were key to matching needs and resources before product development began. First, developers employed the technique of systems engineering to identify gaps between resources and customer needs before committing to a new product development. Second, customers and developers were flexible. Leeway existed to reduce or defer customer needs to future programs or for the developer to make an investment to increase knowledge about a technology or design feature before beginning product development. Third, the roles and responsibilities of the customer and the product developer were matched, with the product developer being able to determine or significantly influence product requirements. In cases where these factors were not present at program launch, product development began without a match between requirements and resources. Invariably, this imbalance favored meeting customer needs by adding resources, which resulted in increased costs and later deliveries.

section_title: Letter
paragraphs: As you requested, this report examines how best practices offer improvements to the way the  Department of Defense defines and matches weapon system requirements to available resources such  as cost, schedule, and mature technologies. It also examines the importance of the timing of this  match and identifies practices that can improve the prospects for making the match before starting an  acquisition program. We make recommendations to the Secretary of Defense on how to better align  the requirements setting and program approval processes to infuse more knowledge earlier into each  process.
We are sending copies of this report to the Honorable Donald H. Rumsfeld, Secretary of Defense; the  Honorable Gregory R. Dahlberg, Acting Secretary of the Army; the Honorable Robert B. Pirie, Jr.,  Acting Secretary of the Navy; the Honorable Lawrence Delaney, the Acting Secretary of the Air Force;  the Honorable Mitchell E. Daniels, Jr., Director, Office of Management and Budget; and to interested  congressional committees. We will also make copies available to others upon request.
If you have any questions regarding this report, please call me at (202) 512-4841. Other key contacts  are listed in appendix II.
subsections: section_title: Introduction
paragraphs: For several years, the Department of Defense (DOD) has expressed an  urgent need to acquire new weapon systems to replace its force that it  believes is becoming outdated and too costly to operate. DOD’s annual  weapon system investment has increased from about $90 billion 3 years  ago to almost $100 billion for fiscal year 2001; over the next 5 years, DOD  plans to spend about $516 billion developing and acquiring weapon  systems. DOD would like to get the most out of this investment, and it has  set goals to develop new weapons in half the traditional time and within  budget. Historically, DOD has not received a predictable return on its  investment in major weapon systems as they have cost significantly more  and taken much longer to complete than originally estimated. When one  program runs into problems and needs more money than planned, it comes  at the expense of delaying or canceling other programs, which reduces  buying power and means less overall modernization. The ability to execute  a program more predictably within cost and schedule estimates would  lessen the need to offset cost increases by disrupting other programs. DOD  recognizes that changes are necessary to its acquisition practices to  achieve its modernization goals. Thus, it has advocated adopting the  practices of leading commercial firms.
Our reviews over the past 20 years have likewise pointed to a need to adopt  new practices. We have seen many of the same problems recur in weapon  system programs—cost increases, schedule delays, and performance  problems. On many occasions, we found that programs required more  resources—time and money—than were estimated for demonstrating  technologies, designing solutions, and providing more production  capabilities in order to meet customer expectations. Because customer  expectations for the system’s performance were set when the decision was  made to invest in the system, adding resources became the primary option  for solving problems when they arose. Despite good intentions and some  progress, our ongoing reviews of DOD’s weapon system acquisitions show  that these persistent problems remain. As a result, we undertook a body of  work that examines weapon system acquisitions issues from a different,  more cross-cutting perspective—one that draws lessons learned from the  best commercial product development efforts to see if they can be applied  to DOD weapon system developments. In past years, leading commercial  firms have developed increasingly sophisticated products in significantly  less time and at lower costs.
Our past work has shown that leading commercial firms expect their  program managers to deliver high quality products on time and within  budget. Thus, the firms have created an environment and adopted practices  that put their program managers in a good position to succeed in meeting  these expectations. Collectively, these practices comprise a development  process that is anchored in knowledge. The firms demand—and receive— specific knowledge about a new product at key junctures in the process,  (see fig. 1).  cost, schedule, and There is a synergy in this process, as the attainment of each successive  knowledge point builds on the preceding one. Such a knowledge-based  process is essential to commercial firms getting better—and predictable— cost, schedule, and performance outcomes. It enables decisionmakers to  be reasonably certain about critical facets of the product under  development when they need it. We have found that when DOD programs  have employed similar practices, they also experience good outcomes. This  knowledge can be broken down into three knowledge points:   when a match is made between the customer’s needs and the available   when the product’s design meets performance requirements, and   when the product can be produced within cost, schedule, and quality  targets.
The most important knowledge point occurs at launch—the point at which  the product developer makes a decision to commit (or invest) the  resources necessary to develop a new product that will meet customer  needs. This knowledge point makes it easier to reach the remaining two  knowledge points at the right time. Successful programs are launched only  when a product developer is confident that it has the resources— technology, engineering, and production knowledge, along with sufficient  time, and money—to develop a product the customer wants. Significant  problems have occurred during development when programs were  launched without this match.
We have reported on how a key resource of a developer—advanced  technology—can and must be readied to meet product requirements at the  time a product’s development program is launched. In this report, we  address both sides of the match: how customer needs and product  developer resources can be managed so that a product developer can  predictably deliver a product the customer wants.
subsections: section_title: How Product Requirements Are Set Is Key to Program Outcomes
paragraphs: The decisions that are made in translating the ideas for a new product into  actual features and characteristics dictate the amount of resources— knowledge, time, money, and capacity—that will be necessary to bring the  product to market. Thus, they may be the most highly leveraged of all  product development decisions. A product’s requirements are based on  customers’ expectations and justify the developer’s investment of  resources to provide the desired capability. Requirements drive the amount  of capital, time, expertise, and technologies the developer must invest. In  the past, it has not been unusual for weapon system requirements to be set  at such a high level that the initial estimate of the resources necessary to  develop a responsive product proves insufficient, evidenced by cost growth  and schedule slippage. The case to justify the requirements is often so  stridently made that decisionmakers are in a relatively weak position to do  anything other than find more resources.
For commercial firms and DOD, the basic process for formulating a  product’s requirements is the same. Each begins with understanding the  customers’ expectations. These expectations are then translated into  product requirements that include the job the product is to perform, the  functions or characteristics it is to possess, the practicality it must have,  and its reliability. Typically, the first understanding of customer  expectations exceeds what the developer can do within available  resources, because the developer has a limited amount of resources at its  disposal for product development. On one hand is knowledge—the  technology and capabilities the developer has to engineer and manufacture  the product. On the other hand is the time and money the developer has to  develop additional knowledge, if need be, and to design, build, test, and  deliver the product. It is not unusual for a customer to want a  high-performing product that does not cost much or take too long to  develop. Such an expectation may exceed the developer’s technology or  engineering expertise or may be more costly and time-consuming to create  than the customer is willing to accept. The developer must stay within its  means if the venture is to remain mutually beneficial. Table 1 characterizes  the divergent interests of the customer and the product developer.
Given these different interests, a customer’s wants and a product  developer’s available resources must be matched to form an achievable set  of product requirements. On one hand, the product developer must develop  and produce the product within the time frames the customer needs or the  customer may find an alternative product or source. On the other hand, the  customer must not demand a product that requires so much money or time  to develop that it cannot be afforded or delivered when needed. There is a  delicate balance that must be achieved between these two divergent  interests before a product can be successfully developed and produced.
On all product developments, there is an attempt to match expectations  with available resources to define the new product. A customer’s  expectations and a product developer’s resources are more closely  scrutinized during the matching process that attempts to bring the two  together. The outcome is a set of product requirements that represent an agreement that the product will meet the customer’s wants and that the  developer can deliver the product within acceptable cost and schedule  estimates. The requirements then guide the development program. This  basic requirements setting process is illustrated in figure 2.
The process of translating general customer expectations into a specific set  of product requirements involves information gathering, analysis,  negotiation, and agreement. In the commercial process, the customer and  the product developer negotiate requirements, matching expectations and  available resources into a documented set of product requirements prior to  committing resources to product development. During this negotiation, the  customer’s relatively unconstrained wants are often reduced to a set of  performance characteristics that are achievable with available resources,  yet still meet the customer’s needs. The commercial process is a two-way  communication between the customer and the product developer. For  example, an airline company may want a certain speed to maximize  revenue per passenger mile from a new aircraft. However, the product  developer may determine that the resources to develop an aircraft with that  speed are not available or must be increased dramatically. Both parties  then work through an iterative process of trades and negotiation to settle  on an aircraft with mutually acceptable performance and resource  requirements.
The DOD process is somewhat more complex and involves  communications among at least four major players. On one end is the  customer, which is normally a military organization that belongs to a major  fighting force. On the other end is the product developer, usually a defense  firm that serves as the prime contractor for developing and producing the  weapon system. In between are two other players that actually negotiate  needs and resources to arrive at product requirements. One is referred to  as the user representative, which is an organization separate from the  customer but represents the customer and negotiates on its behalf. The  other player is the DOD program manager, a separate organization that, in  essence, represents the product developer. Figure 3 illustrates how these  different players interact in commercial and DOD requirement-setting  processes.
Both commercial and defense organizations are concerned about how  much a product or weapon system is going to cost, how long it is going to  take to build, what resources will be needed to build and maintain it, and  whether it works properly. All of these concerns are translated into the  product’s requirements. Unlike the commercial process, the DOD product  developer does not directly influence the product requirement prior to  launching product development. Once requirements are formalized in what  DOD refers to as the Operational Requirements Document, they are turned  over to the prime contractor, who actually begins product development.
section_title: Objectives, Scope, and Methodology
paragraphs: The Chairman and the Ranking Member, Subcommittee on Readiness and  Management Support, Senate Committee on Armed Services, requested  that we examine various aspects of the acquisition process to determine  whether the application of best practices can improve program outcomes.  To date, we have issued reports on advanced quality concepts, earned value  management, management of a product from development to production,  management of key suppliers, management of technology insertion,  training, and management of test and evaluation (see related GAO  products).
This report covers the beginning of the acquisition process: the  management of product requirements. Our overall objective was to  determine whether best practices offer methods to improve the way DOD  sets product requirements within the framework of a knowledge-based  product development process. Specifically, we assessed (1) the effect of  the timing of the match between the customer’s needs and the developer’s  resources on a product’s cost and schedule; (2) the best practices for  obtaining this match during the requirements setting process, compared  with more traditional DOD practices; and (3) the progress made and  challenges DOD faces in adopting best practices for setting requirements  on individual weapon systems.
We follow a similar overall methodology for conducting best practices  reviews in the area of weapon system development. We start by identifying  individual aspects of weapon system development—in this report, the  setting of requirements—that have been shown to be a significant and  recurrent cause of problems. Our sources for such information include our  many reviews of individual weapon systems; studies from other sources,  such as the Defense Science Board; and discussions with defense experts,  including past and current DOD officials, defense industry representatives,  and analysts from private organizations that study defense issues. Before  beginning a review of a particular topic, we confirm with DOD officials that  the topic is one in which the potential for improvement is significant. Once  we have identified the topic, we use a case study approach because case  studies provide the in-depth knowledge needed to understand individual  practices, how they affect program outcomes, and why they are adopted. In  selecting case studies, we look for examples of excellent practices from  leading commercial firms, examples of typical or traditional practices from  DOD, and where possible, DOD examples that exhibit excellent practices.  In making our selections, we are careful to make sure that there is a link  between the practices themselves and the outcomes of the programs.
To identify best practices for setting requirements on commercial products,  we reviewed literature and spoke with industry and academic experts to  find companies recognized for managing requirements to help deliver new  products that were both quicker to market and more advanced than their  predecessors. We identified three companies   Caterpillar Construction and Mining Division, Decatur, Illinois;  Bombardier Aerospace, Toronto, Canada; and  Bethlehem Steel, Bethlehem, Pennsylvania We visited each company, discussed the process used for setting  requirements, and obtained an understanding of the overall process used  with emphasis on those practices each felt were critical for success. We  also met with individual program managers and discussed specific product  development examples that further illustrated the process. During our  discussions with the firms, we compared and contrasted the best practices  with DOD’s practices.
We developed nine case studies in total. These included two commercial  case studies, one National Aeronautics and Space Administration (NASA)  program that had received excellent results by disciplining its  requirements-setting process, and six DOD weapon system programs that  represented a mixture of traditional and best practices. We reviewed the  requirements-setting process for all nine programs. For each program, we  interviewed key managers and obtained documentation to determine  (1) the process that was used to achieve the match between customer  wants and resources to form the product’s requirements, (2) the timing of  this match and the tools used to achieve it, and (3) the extent to which the  requirements setting process affected the program’s product development  outcome. Descriptions of the nine programs we reviewed follow.
Bombardier’s BRJ-X, a commercial jet in development, designed to carry  between 88 to 110 passengers. It bridges the gap between the current  fleet of regional jets, 20 to 70 passenger capacity and the larger 111 to  170 passenger commercial airlines. The BRJ-X program was launched in  the second quarter of 2000. The first aircraft is scheduled to be delivered  to airlines in late 2003. Bombardier estimates customers for 2,500  aircraft over the next 20 years.
The Caterpillar 797 mining truck, the largest mining truck ever built. It  can carry over 360 tons of ore, and features many patented innovations.  Developed in response to mining companies’ desire to reduce cost per  ton of hauling ore in large-scale mining operations, Caterpillar launched  the 797 program in 1997.
The Air Force’s Global Hawk, an unmanned aircraft that is intended to  fly at altitudes as high as 65,000 feet and for as long as 40 hours to  provide the Air Force with an intelligence, reconnaissance, and  surveillance capability. The Air Force built five prototype technology  demonstrators that were used to demonstrate the aircraft, and it plans  to launch the product development program in 2001.
The Army’s Crusader artillery vehicle program, a self-propelled 155-millimeter howitzer and resupply vehicle. It is expected to be the  first fully automated, computerized, and tracked artillery system. The  Crusader development program began in 1994, and the howitzer is  expected to start production in 2008. The development program is  estimated to cost $4.3 billion.
The Army’s Tactical Unmanned Aerial Vehicle, a short-range unmanned  aircraft that is expected to provide the Army with day or night  reconnaissance, surveillance, and target acquisition capability. The  Army began development in March 1999. It plans to buy 44 systems  starting in 2001. Each system includes three unmanned aircraft; a  vehicle to carry the aircraft; two ground control stations mounted on  vehicles; and launch, recovery, and support equipment pulled on trailers  behind the vehicles. The cost to buy the 44 systems is estimated at  $430 million through 2004.
The Navy’s Radio Frequency Countermeasures system, an electronics  warfare system that uses a jamming device called a techniques  generator. It is carried onboard an aircraft to produce jamming signals  that are transmitted by fiber optic cable to a towed device that acts as a  decoy for the aircraft. The system is used to protect the aircraft from  radar-controlled weapons like missiles and antiaircraft artillery. It is a  critical component of the Integrated Defensive Electronics  Countermeasure System being developed for some Navy and Air Force  aircraft. System development began in 1995 and is expected to cost over  $200 million. It is expected to enter production in 2002.
The Army’s Brilliant Anti-Armor Submunition program, referred to as  BAT, an acoustic and infrared terminally guided submunition that  searches for, detects, tracks and engages moving tanks and armored  combat vehicles. Its mission is to provide deep attack against motorized  rifle and tank divisions. The carrier for the submunition is the Army  Tactical Missile System that is launched from the Multiple Launch  Rocket System. The Army plans to buy 15,707 submunitions at a cost of  $2.5 billion.
The Army’s Comanche helicopter, a lightweight, twin engine, stealthy  helicopter that is intended to replace the Army’s OH-58 and AH-1  helicopters. The primary mission of the aircraft will be armed  reconnaissance and attack. It is the Army’s largest aviation acquisition  program, with a projected total development and production cost of $48 billion for 1,213 helicopters. The development program was  launched in 1988, and production is expected to start in 2006.
The Far Ultraviolet Spectroscopic Explorer (FUSE), a scientific  telescope that is used for studying the origin and evolution of stars,  galaxies, and planetary systems. The telescope is 18 feet tall and weighs  3,000 pounds. It was developed for NASA by the John Hopkins  University. The FUSE program began in June 1996 and was completed in  June 1999 at a cost of $120 million.
We used information from our prior best practices work, including most of  the information on the BAT and Comanche programs. Similarly, we  gathered knowledge about many aspects of the product development  processes, including the setting of requirements, from leading commercial  firms in addition to the firms included in this report. During the past 4 years, we have gathered information on product development practices  from 3M, Boeing Airplane Company, Chrysler Corporation, Ford Motor  Company, Hughes Space and Communications, and Motorola. This  information enabled us to develop an overall model to describe the general  approach leading commercial firms take to developing new products.
We also met with experts in the area of setting product requirements from  academia and participated in conferences and workshops with recognized  leaders in the acquisition field to obtain information on how organizations  were improving their acquisition processes. To obtain a general  understanding of DOD’s requirements setting process and improvement  initiatives, we met with officials from the Office of the Secretary of  Defense; Army, Navy, and Air Force Headquarters; and the Joint Staff for  the Joint Chiefs of Staff. We also had discussions with former DOD officials  and industry experts about DOD acquisition policies and practices. With  these officials we discussed the current process, initiatives and the  applicability of best practices to DOD operations. In addition, we visited  NASA to obtain information on their processes and practices for setting  requirements for new product development programs.
We conducted our review from November 1999 through December 2000 in  accordance with generally accepted government auditing standards.
section_title: Timely Matching of Requirements and Resources Is Critical to Product Development Outcomes
paragraphs: The point in time that a developer for a new product becomes justifiably  confident that it has the resources—knowledge, capacity, time, and  money—to develop a product that a customer wants is critical to the  success of the development effort. Barring program cancellation, the match  between resources and wants is eventually met on just about every product  or weapon system development. A key distinction between successful  products—those that perform as expected and are developed within  estimated resources—and problematic products is when this match is  achieved. Simply put, we found that when wants and resources were  matched before a product development was started, the more likely the  development was able to meet performance, cost, and schedule objectives.  When this match took place later, programs encountered problems such as  increased costs, schedule delays, and performance shortfalls.
For successful product or weapon system development program cases,  trade-offs were made either in the design of the product or in the  customer’s expectations to avoid immature technologies or exotic  components that threatened to outstrip the developer’s resources. In the  less successful cases, the opportunity to make such trade-offs before  starting product development was missed either because gaps between  expectations and resources were not identified or because customers were  unwilling to reduce their expectations. When the divergence between  customer expectations—which by then had become firm product  requirements—and developer resources was recognized and confronted,  decisionmakers were reluctant to materially change the requirements.  Consequently, the developer had to invest more resources than originally  planned to meet the requirements.
subsections: section_title: Early Matching of Customer Expectations and Developer’s Resources Is Critical to Program Success
paragraphs: We found that the timing of the matching process—when the customer’s  expectations were successfully matched with the product developer’s  resources—significantly influenced the likely success of a product’s  development. After reviewing the process for defining product  requirements for nine development programs, we found a relationship  between when expectations were matched with available resources and  when the cost and schedule predictions for the programs were achieved.  This relationship is shown in table 2.
The more successful programs had matches before the commitment to  launch the programs was made. In each case, the product developer had  done the initial design of the product and ensured that only proven  technologies, design features, and production processes would be used.  This was accomplished by either making additional investments to  demonstrate uncertainties such as new technology or by reducing the  product’s initial performance requirements. These steps maximized the  knowledge content of the product and enabled the program manager to set  cost and schedule estimates it could reasonably expect to meet. In  contrast, the programs that did not have matches before launch did so  during product development by the unplanned addition of resources. This  contributed significantly to cost and schedule problems. Figure 4 illustrates  the timing of the match between customer’s expectations and a product  developer’s resources.
The programs in which expectations and resources were matched before  product development started were in a good position to commit to cost and  schedule estimates that were attainable. In those cases where expectations  and resources were not matched before launch, cost and schedule  estimates had to be made at the time of launch. Such estimates were  necessarily made at levels consistent with the resources the product  developer had available, under the assumption that either (1) no gaps  existed between expectations and resources or (2) any gaps could be  closed within projected resources. Because customer expectations  regarding the performance of the product tended to become set when  product development began, adding resources emerged as the primary  option available to match expectations and resources. These resources  (time and money) were typically needed for maturing technologies,  developing design solutions, and providing more production capabilities.  Perhaps more importantly, they were not estimated or planned for and  often necessitated sacrifices in other needs, such as reducing the resources  of other development programs.
section_title: Trade-offs Are Critical to Matching Customer Expectations With Developer Resources Before Starting Product Development
paragraphs: Caterpillar and Bombardier both matched customer expectations with  available resources prior to setting product requirements and launching the  products’ development programs. In each case, there were differences  between expectations and resources that necessitated trade-offs before  requirements could be set and the product’s development could be  launched. For these cases, expectations and resources were negotiated so  that the product’s requirements could be achieved within available  resources while still meeting the customers’ critical needs. This allowed  the firms to develop and deliver their products quickly and within  acceptable cost limits, thereby maintaining their competitive advantage in  their respective markets.
subsections: section_title: Caterpillar’s 797 Mining Truck
paragraphs: To maintain the competitive advantage in its market, Caterpillar’s board of  directors believed it had to develop by the end of 1998 a new product that  was sized to work efficiently with large loading shovels used in mining  operations. This meant that Caterpillar’s product development team would  have about 18 months from the time the product development program was  approved to develop and field the 797, a newly designed truck that could  efficiently haul at least 360 tons of payload. According to Caterpillar, they  met this date because they made trade-offs between the customers’  expectations and the resources available before the product development  program began. Figure 5 shows the 797 mining truck developed and figure 6 shows the history of the truck development.
Program launched after requirements that could not be met with available not match resources. Trade-offs needed to technologies were deferred to future was within 5 percent of its cost objective. achieve18-month schedule. versions of the 797.
Examples of key trade-offs Caterpillar made to close gaps between  customer expectations and its own resources were:   deferring to the next product line, new prognostic technologies wanted  by the customer that could assist in forecasting wear and tear to the  truck, but were immature at the time product development started;  selecting a twin-engine propulsion design to power the 797 rather than a  single engine—despite its potential for lower operating costs—because  it had not yet been developed and was therefore too much of a risk; and  redesigning the wheel and transmission to avoid the need to develop  new gears for the differential unit in the drivetrain, which avoided a  costly and risky development effort that could have impacted the truck’s  development progress.
section_title: The Bombardier BRJ-X Jet
paragraphs: During the BRJ-X jet’s concept definition phase, which preceded product  development, Bombardier identified some customer expectations that its  designers believed would put program cost and schedule at risk. The  expectations were analyzed and trade-offs were made to make the design  achievable within resources. For example, both airline customers and  Bombardier wanted to use fly-by-wire flight control technology, which  would replace heavy hydromechanical flight controls with lighter weight  electronic controls, on the BRJ-X to reduce weight and lower fuel costs.  However, Bombardier engineers had some concerns about the technology  since it had never been integrated into a regional jet configuration before.  Consequently, Bombardier decided to invest the time and money to  demonstrate the technology on a surrogate business jet before making it a  program requirement. Another trade-off was made after Bombardier  determined that the desired speed would require design effort and features  not fully proven out. The customers agreed that a slight reduction in speed  would eliminate the design concerns yet still meet their expectations.
section_title: NASA’s FUSE Program
paragraphs: FUSE is a telescope used to study the origin and evolution of certain  elements in space to help determine the age and evolution of stars,  galaxies, and planetary systems. The customer initially expressed  expectations in terms of gathering precise images of light from these  elements in space 24 hours per day. When NASA threatened to cancel the  program because requirements were not achievable given available  resources, FUSE program managers negotiated with customers and found  they could still meet the basic expectations with reduced requirements for  a high-resolution mirror, bandwidth detection, and time on orbit; switching  from a highly elliptical orbit to a low earth orbit (see fig. 7).
These reductions in requirements allowed the use of existing technologies  such as a new grating technique to spread radiation into different  wavelengths. Matching requirements to resources allowed FUSE to not  only meet schedule targets and be within 20 percent of its cost objective  but also a critical NASA need.
section_title: When Matching Did Not Occur Before Program Launch, Developers Were Forced to Add Unplanned Resources
paragraphs: In cases where the customers’ expectations were not matched with the  developers’ resources before product development began, the matches  took place after opportunities to make trade-offs had passed. Because new  programs had been approved with customer wants formally documented as  requirements, the main avenue available to close gaps between  requirements and resources was for the developers to invest more effort,  time, and money to gain the knowledge and capacity needed to meet the  requirements. This additional investment—manifested by cost increases  and schedule delays—was not planned, which forced trade-offs or cuts to  be made in other programs to free up the additional resources. In some  cases, the developers may have had indications that there was a mismatch  at the launch decision, but they were pressured to go forward anyway. In  other cases, while the developers may not have enough knowledge to be  confident that there was a match before launch, they were at a  disadvantage to argue that there was not. DOD’s Radio Frequency  Countermeasures system and Comanche helicopter programs did not have  matches before their product development programs were launched. In  fact, despite being several years into both programs, it is still uncertain  whether this match has been reached.
subsections: section_title: Navy’s Radio Frequency Countermeasures System
paragraphs: Customer expectations and developer resources were not matched on the  Radio Frequency Countermeasures program when product development  began in 1995. Essentially, the performance wanted by the customers  exceeded the time, money, and technologies available to the product  developer to develop an acceptable system. For example, a critical  component is the fiber-optic towed decoy—a component, towed behind the  aircraft in flight, which transmits electronic countermeasures. Before  program launch, the requirement for the power to be transmitted from the  towed decoy was nearly tripled because of a last minute addition to satisfy  the Air Force’s expectations for the F-15 fighter and the B-1 bomber. This  was a demanding requirement that, according to the former program  manager, required some technological invention. According to the manager,  this was complicated by the customer’s refusal to either reduce  performance requirements or accept increases in time and cost to develop  a product that would meet these requirements. The Navy chose to launch  the program with this mismatch, recognizing that it added risk to the  program. Figure 8 shows the timeline for the program.
Developer reports 197% cost increase and 15-month schedule delay As the program proceeded, problems associated with the risk  accompanying the mismatch turned into development problems, and the  additional resources found to be unacceptable to the customer before  launch were accepted as a necessity after launch. Consequently, the costs  rose from $74 million to $221 million and the development schedule was  extended 15 months.
section_title: Comanche Helicopter
paragraphs: Customer expectations that became key requirements for the Comanche  helicopter demanded several technologies that were still very immature  when the Army decided to launch the program in 1988. For example, the  integrated avionics and an advanced infrared night vision and targeting  sensor were included on the program when they were still conceptual in  nature. These advanced avionics systems were needed to meet the  customer’s “must-have” requirements for a very lightweight, stealthy, highly  maneuverable, all-weather reconnaissance and attack helicopter. These  technologies were also critical to meet cost and weight goals for the  program. The Army launched the program despite the low readiness of the  technologies, with the developer having limited design alternatives but  believing that the needed technological invention could be accomplished  within projected resources. At the time of launch, the Army estimated that  product development would cost $3.6 billion and last about 8 years. Due to  problems that developed with these technologies and budgeting and other  changes in the program, development is now estimated to take $8.3 billion  and 18 years—over 100-percent increases. The Army kept the customer’s  requirements essentially unchanged, electing to double the resources  needed to meet them.
section_title: Several Factors Enable Customer Wants and Developer Resources to Be Matched Before Program Launch
paragraphs: The ability to match a customer’s wants with resources before launching a  development program is key to putting program managers in a better  position to succeed. We found three factors that comprise this ability. First,  developers employed systems engineering to identify gaps between  resources and customer wants before committing to a new product  development. Second, customers and product developers were flexible  before launch. Leeway existed to reduce expectations, defer them to future  programs, or to invest more resources up front to eliminate gaps between  resources and expectations. Third, the roles and responsibilities of the  customer and the product developer were balanced, with the product  developer given the responsibility to determine or significantly influence  product requirements. In cases where these factors were not present at  program launch, product development began with imbalanced product  requirements. Invariably, this imbalance favored meeting expectations at  the expense of resources, putting the developers at a disadvantage to  deliver the products within cost and schedule estimates.
In the most successful cases, the effective interplay of these factors  allowed the customers and product developers to arrive at a set of product  requirements that could be developed within cost and schedule targets.  Systems engineering provided knowledge necessary to translate customer  wants into specific capabilities, enabling the developers to identify and  resolve gaps before product development began. With systems engineering  knowledge in hand, flexible requirements were essential to lowering risk  through negotiations because knowledge alone did not produce trade-offs.  Absent such flexibility, resources and wants could still be matched before  product development began, but the options to resolving any gaps were  limited to additional investments on the developers’ part. Finally, with  knowledge gained from systems engineering and flexible requirements as  preconditions, successful product development programs benefited from  an environment in which product developers and customers had balanced  roles and shared responsibilities for setting product requirements.
subsections: section_title: Systems Engineering Tools Are Critical for Identifying Gaps Between Developer’s Resources and Customer’s Expectations
paragraphs: When product developers employed systems engineering before  committing to product development, they were able to identify areas in  which the customers’ wants exceeded their resources. For those cases in  which developers did not conduct sufficient systems engineering before  committing to the new product development, they were weakened in their  ability to identify gaps between their resources and expectations. These  gaps were later revealed as unexpected problems that required invention,  time, and money to resolve.
Systems engineering is a process that not only translates customer wants  into specific capabilities, such as individual technologies and  manufacturing processes, but also provides knowledge that enables a  developer to identify and resolve gaps before product development begins.  It is defined as a logical sequence of activities that transforms a customer  want into specific product characteristics and functions and ultimately into  a preferred design (see fig. 9). It is not necessarily the use of systems  engineering in the development of a new product or weapon system, but  when it is used that distinguishes it as a best practice. of the requirements into a set of specific technical and design solutions needed to system must perform. functions.
The systems engineering discipline enables the product developer to  translate customer wants into specific product features for which requisite  technological, software, engineering, and production capabilities can be  identified. Once these capabilities are identified, a developer can assess its  own capabilities to determine if gaps exist. It is critical for a developer to  involve the right people—those with the affected areas of expertise—in  this assessment. Gaps identified between what the customer’s wants are  and what the developer possesses then become the focus of analysis. Some  gaps can be resolved by investments the developer makes, while others can  be closed by finding technical or design alternatives. Remaining gaps— those that represent capabilities the developer does not have or cannot get  without increasing the price and timing of the product beyond what the  customer will accept—must be resolved through trade-offs and  negotiation.
During systems engineering, a product design progresses through at least  three iterations. The first is a notional design—a general concept of what  the product will look like and what it might be capable of that is  unconstrained by resources. The second iteration is the first detailed  design that enables a developer to compare its capabilities with the  demands of the product. The third iteration is the final design, which  captures improvements to the design generated by testing, analysis, and  other forms of learning. These iterations can be seen in figure 10, which  provides a general comparison of the amount of systems engineering  accomplished in the successful and problematic cases prior to launching a  product development program.
subsections: section_title: Successful Programs Used Systems Engineering Before Product Development Began
paragraphs: In each successful case, the product developer worked closely with the  customer to understand its wants, which were often articulated in a  bottom-line metric, such as cost per passenger mile for a commercial  airplane. By the time the developer had committed to a product  development program, it was well into its systems engineering process and  had developed a preliminary design of the product. This process identified  the gaps between resources and expectations, which could be then  addressed through investments, alternate designs, and, ultimately,  trade-offs. The knowledge produced by the process put the developer in a  good position to negotiate with the customer because consequences could  be associated with attempts to meet those wants that exceeded the  developer’s capabilities. The process also involved the customer through  periodic reviews and acceptance of the product’s final design. According to  some commercial representatives, systems engineering is a good  investment to reduce risk, usually comprising a small percentage to the  overall development cost of a new product. We found two commercial  firms—Caterpillar and Bombardier Aerospace—and two DOD programs— the Army’s Tactical Unmanned Aerial Vehicle and the Air Force’s Global  Hawk unmanned aerial vehicle—that employed fairly extensive systems  engineering before they committed to product development.
subsections: section_title: Caterpillar’s 797 Mining Truck
paragraphs: Caterpillar’s New Product Introduction process calls for completing a  significant amount of the systems engineering process during the concept  phase for any new product before product development. It applied this  process to its development of the 797 mining truck, a new product design.  It spent significant time and effort prior to beginning product development  establishing the customers’ wants and closing the gap between them and  available resources. Before committing to its development, Caterpillar  gathered information from the mining companies about operating  conditions and the cost per ton of ore hauled desired of the new truck that  they would be willing to accept during the truck’s lifetime. Caterpillar then  made a preliminary determination of what the truck’s performance  requirements would have to be to meet these conditions and costs. For  example, it determined that the truck’s payload, or bed, would have to haul  at least 360 tons, travel at certain speeds, and climb certain grades. This  information served as the starting point for systems engineering to  determine if the performance requirements for the 797 were achievable  given Caterpillar’s resources. Once these requirements were established,  Caterpillar’s engineers began an iterative systems engineering process to  design a product that would meet the customers’ wants and could be  developed within funding, schedule, and resource targets. This process  culminated in a specific product solution that matched requirements to  resources before Caterpillar committed to product development.
The systems engineering process for the 797 forced key trade-offs between  performance requirements and design solutions prior to commitment to  product development. For example, to transmit the power from the engine  to the rear wheels, the original design called for using very large differential  gears. Upon reviewing that design, an experienced Caterpillar production  engineer noted that no gear manufacturer made a gear that large and that to  create such a production capability would be risky. Consequently, the  design engineers found an alternative that called for making incremental  changes in the transmission and wheel designs, which enabled existing  differential gears to be used. These design changes and the processes  required to make the changes were demonstrated successfully prior to  product development. Systems engineering also revealed risk on some of  the new hydraulic technology for the 797 that would allow easier and more  reliable unloading. Because this new technology had not been used in the  field before, Caterpillar engineers demanded demonstrations and field tests  of the technology before allowing it onto the 797 truck.
section_title: Bombardier’s New Regional Jet
paragraphs: Bombardier set requirements for a new, larger family of regional jets that  will carry up to 115 passengers—the BRJ-X series—using its Bombardier  Engineering System. Beginning with an overall cost-per-passenger-mile  target from the customer, Bombardier employed an iterative systems  engineering process to make trade-offs between performance  requirements—which were based on extensive market analysis—and  available resources to arrive at requirements, design, and cost and schedule  targets before committing to product development. The company plans to  achieve nearly full knowledge of the product’s design before it commits to  product development. At launch, the product’s performance requirements  and its configuration will be frozen.
During Bombardier’s trade study process, customer wants are thoroughly  challenged by the firm’s assessment of technology capability and readiness,  as well as by manufacturing, producibility, logistics, and other implications  of the product’s features. During this process, engineers are cognizant that  product success and profitability hinge on not only product features but  also product price and quantity that market research has determined. This  fact tempers decisions about the viability of key performance or design  parameters. In other words, if Bombardier cannot produce and develop a  product within a customer’s price range, then the customer is not likely to  buy it. Once this process is complete, Bombardier performs a detailed  aircraft review, which results in a trade-off analysis. If this analysis  suggests that key performance requirements—speed or range, for  example—cannot be achieved within the established price and quantity, the  requirements are changed. If product performance is reduced below the  customer’s minimum threshold, profit calculations are upset and the  development program is not initiated.
section_title: Army’s Tactical Unmanned Aerial Vehicle
paragraphs: The Army’s Tactical Unmanned Aerial Vehicle is a DOD program that used  systems engineering to gather knowledge prior to starting development.  The vehicle must meet the brigade commanders’ need for a day or night,  adverse weather, multisensor data collection system with improved  communication with joint forces that provides real-time battle information  and cannot be observed by the enemy. The program grew out of an  Advanced Concept Technology Demonstration program that was started in  1996 and completed in 1998. While the air vehicle that was flown and  evaluated during this demonstration met most of the Army’s close-range  reconnaissance requirements, it did not meet all requirements and was  canceled. Figure 11 shows the Army’s Tactical Unmanned Aerial Vehicle.
The technology demonstration provided the Army with knowledge about  the achievability of the customers’ requirements. The product developer  built a prototype system that was used to identify gaps between what the  customer wanted and what was achievable with resources. Through  systems engineering, the developer transformed the wants into a set of  performance requirements that led to the prototype’s design. The Army  used the results of the technology demonstration to define the customer’s  requirements that were geared toward obtaining a system that required  minimal development, based largely on what was demonstrated. Because  of the knowledge gained from systems engineering, the Army was able to  stage a fly-off of four prototype designs within 9 months after the product  development program began. The fly-off results provided additional  knowledge to help match customer wants with the product developer’s  resources through further trades. The Tactical Unmanned Aerial Vehicle  that is now in development closely resembles the design that won that  competition. Another program, the Air Force’s Global Hawk, followed a  similar path by conducting systems engineering and building prototype  systems prior to starting its program.
section_title: Problems Arose When Only Limited Systems Engineering Was Done Prior to Launch
paragraphs: In the problematic cases, product developers had not progressed as far into  the systems engineering process at the time the program was launched. In  most of these cases, the product developers had only notional designs at  that point—not thorough enough to translate expectations into specific  functions against which resources could be compared. It was not until  product development was underway that systems engineering was fully  employed to create a preliminary product design. In each case, this  occurred several years after the acquisition program was started. The lack  of knowledge at the start of each program made well-informed trades  between customer wants and developer resources difficult to see or make.  Nonetheless, cost and schedule targets for each program were set based on  available information. Problems that were discovered during product  development and during the systems engineering process often resulted in  the need for more time or money than had been estimated at program  launch.
Two DOD programs we reviewed—the Radio Frequency Countermeasures  system and the Crusader program—initiated a systems engineering process  before product development; however, it was not extensive and performed  by the eventual product developer hired to design the product and only  resulted in a notional product design prior to the decision to commit to  product development. The product developers did not gain significant  knowledge about what was possible, given the availability of resources,  until after the decision was made to commit the resources toward  developing the product.
subsections: section_title: Navy’s Radio Frequency Countermeasures System
paragraphs: The majority of the systems engineering by the Radio Frequency  Countermeasures product developer—the defense firm that was awarded  the development contract—was not done until after the Navy had launched  the program. The knowledge used to match requirements with funding,  schedule, and other resources was limited before the program was  launched. The Navy did enough analysis—about 25 percent of systems  engineering, according to the former program manager—to form a notional  design of the system. The notional design was based on a top-level analysis  of the functions, which was done primarily by the Navy’s program office.
The Navy nonetheless considered this amount of knowledge sufficient to  commit to product development.
The Navy entered product development assuming that a large number of  the needed parts would be nondevelopmental items—items already used  on products. The Navy estimated that 90 percent of the parts for one of the  most critical subsystems, the techniques generator, would be mature,  readily available items. According to the program manager, it was not until  the contractor hired to develop the system began the detailed systems  engineering process that it discovered that only about 50 percent of those  parts were still available. Many of the required parts had become obsolete  and had to be replaced with redesigned or newly developed parts— revealing a gap between what was thought to be within the developer’s  capabilities and what the requirements were. As a result, problems with the  assumptions made about the notional design by the Navy were discovered  in product development, after resources had been determined and the  fielding date had been established. This resulted in significant disruptions  to planned cost and schedule; eventually affecting the product’s fielding.
Army’s Crusader Artillery Vehicle The Crusader artillery vehicle is another case in which systems engineering  was not performed by the product developer, to a large extent, until after  the acquisition program had been launched. The development program was  launched based on a notional design done by the Army’s program office,  not extensive systems engineering done by the contractor that would  design and build the product. Key to this notional design was the use of a  liquid propellant—new technology—for firing weapon projectiles. Program  officials stated that the optimal range that the Crusader is required to fire a  weapon is still based on the use of this liquid propellant. The Army  assessed various aspects of the risk of developing the liquid propellant  technology and integrating it into the weapon system between low and  moderately high. Nevertheless, on the basis of the notional design, the  Army committed to launching product development.
After the program was launched in 1994, the product developer was  awarded a contract to develop the Crusader. According to a program  official, it took 2 years of systems engineering to determine if the  requirements were feasible given established cost and schedule targets. In  1996, the product developer determined that the liquid propellant  technology was high risk in all aspects and that it would cost an additional  $500 million to develop. This was much more than originally estimated at  the start of the program and represented a major resource gap. As a result,  the Army elected to use a more traditional, less capable back-up—a solid  propellant—to achieve a match between available resources and  requirements. This decision may impact system performance since,  according to program officials, the Crusader will probably not achieve its  optimal firing range. The firing range and the liquid propellant were part of  the reason for launching the program in the first place. Figure 12 shows the  Crusader artillery vehicle.
section_title: Flexibility in Setting Requirements Is Key to Closing Gaps Between Customer Expectations and Developer Resources
paragraphs: While knowledge is essential to identifying gaps between expectations and  resources, it takes flexibility on the part of both the customer and the  product developer to close the gaps. Flexibility represents the customer’s  ability and willingness to lower product expectations, coupled with the  product developer’s willingness and ability to invest more resources to  reduce technical risks and other gaps before program start. Flexibility,  when informed by systems engineering knowledge prior to program  launch, was essential to lowering risk and reducing the cost and length of  product development because knowledge alone does not produce trade-offs. Absent such flexibility, resources and wants can still be matched  before starting product development, but the options to closing any gaps  exposed by systems engineering are limited to additional investments on  the developer’s part. The scenario with the most potential for costly  problems is one in which neither the requirements are flexible nor  sufficient systems engineering has been done to reveal resource gaps  before launch. Several of the DOD programs we reviewed that had  significant problems in meeting product development objectives followed  this scenario.
While there are instances in which flexibility on the part of the customer is  inherent, such as for a unique product that has no predecessor, in most  cases, potential customers for a new product already have expectations. In  these cases, flexibility has to be fostered. We found two factors that  fostered such flexibility before product development started. First, product  development cycle times were limited. This forced product developers and  customers to agree on requirements that were achievable within  established time limits. Second, the developer enlisted the customer’s trust  through an evolutionary approach to product development, which relieved  the customer of the pressure to want all needs met in a single product  iteration. These factors made customers more willing to defer  requirements that demanded more time or unproven technologies for  succeeding versions of the product. In contrast, when these factors were  not present and trade-offs were not made, there was an implicit decision to  accept the risk of not having the product or the capability when needed.
subsections: section_title: Requirements Were Flexible Until Program Launch on Successful Cases
paragraphs: In successful cases, requirements were flexible until the decision was made  to commit to product development because both customers and developers  wanted to limit cycle time. This made it acceptable to reduce, eliminate, or  defer some customer wants so that the product’s requirements could be  matched with the resources available to deliver the product within the  desired cycle time. The customer had incentives to trade off some wants,  owing to the desire to get the product within cost and time predictions and  to confidence that future versions of the product would meet many of those  wants. The two commercial companies we visited exhibited these  characteristics. In DOD, we found two programs that had exhibited this  flexibility before launch.
subsections: section_title: Caterpillar’s 797 Mining Truck
paragraphs: Because of mutual interest on the part of the product developer and the  customer, Caterpillar’s board of director’s mandate for the new mining  truck to be developed before the end of 1998 became a catalyst for  flexibility in requirements and design decisions. For example, Caterpillar  wanted to introduce new prognostic technology on the 797 that could  significantly lower the cost of operating the truck—an expressed want of  the customer. At the heart of this technology were monitoring sensors that  could assist in forecasting wear and tear on the truck—such as on the  powertrain, brakes, and tires—thereby allowing better maintenance  management that would reduce how often components had to be replaced  and increase the truck’s life. However, Caterpillar engineers concluded that  there was not enough time to make this technology mature enough to be  included in the initial design if the 18-month delivery schedule was to be  met. Despite the desire to have the sensors on the 797, the customer  understood the need to make a trade-off and was willing to do without the  sensor on the initial version if it meant getting the truck on time. As a  result, Caterpillar deferred the technology, despite the increased  performance it offered, because it would put other program resources— most importantly, schedule—at risk. The customer accepted this trade-off.  The trade-off was made possible by Caterpillar’s recognition of the risk, the  customer’s trust in the next generation of the truck, and the mutual desire  to deliver the basic product on time and within cost.
section_title: Bombardier’s New Regional Jet
paragraphs: Bombardier’s goal is to limit cycle time on all product developments to 36 months. At the start of a new product’s development process,  decisionmakers ask, “What can we accomplish within 36 months?”  Bombardier representatives explained that this forces them to have  significant knowledge about a new product’s cost and performance early in  the process and is consistent with an evolutionary approach to product  development. If a product’s requirements force them to consider longer  development time frames, they must consider a future generation of the  product or a new and different product. This time frame played a role in the  decision to reduce the BRJ-X customer’s requirement for speed. The initial  speed requirement, based on market research, was originally .81 mach.  After subjecting the requirement to systems engineering, Bombardier’s  engineers deduced that, while speeds of up to .78 mach could be achieved  with current off-the-shelf propulsion technology, new and undemonstrated  propulsion technology would be required to achieve .81 mach. After gaining  additional knowledge through wind tunnel testing, the engineers concluded  that to design and develop the needed propulsion and aerodynamics would  create significant risk for the 36-month product development time frame.
Before launching the BRJ-X development program, Bombardier’s  marketing staff went back to the customers—over 30 airlines—and  presented the information showing that the additional speed would require  more development time and thus be more costly. On the strength of the  analysis, the customers agreed that .78 mach would still meet their needs.  In fact, they discovered that most European air traffic controllers would  not allow the higher speed. Based on the agreement, Bombardier reduced  the requirement to .78 mach, allowing the use of a low-risk propulsion  system. The 36-month goal forced the developer and the customers to  negotiate between the product’s requirements and resources, while  systems engineering provided the knowledge so that informed trade-offs  could be made.
section_title: Air Force’s Global Hawk and Army’s Tactical Unmanned Aerial Vehicles
paragraphs: In two DOD cases that have met product development objectives so far — the Air Force’s Global Hawk program and the Army’s Tactical Unmanned  Aerial Vehicle program—senior acquisition executives placed resource  constraints on the programs and advocated evolutionary acquisitions,  which provided the customers and product developers the incentives and  the opportunity to cooperate and make mutually beneficial trade-offs. The  Tactical Unmanned Aerial Vehicle had a schedule for first delivery of a  system within 1-½ years after launch and the Army established an  evolutionary acquisition strategy for the product to meet this schedule.  This mandate was driven by a mutual agreement between senior executives  from the Army’s acquisition and requirement setting communities that the  priority is to field it quickly. This agreement fostered a good relationship  between the two communities that allowed a “no bells and whistles”  approach to development.
The product manager told us that using the guidance as a foundation, the  customer agreed to collaborate on a basic list of key performance  parameters as the only “must-have” requirements for the initial Tactical  Unmanned Aerial Vehicle and define the remaining requirements as  tradable against resources, such as cost and schedule. The requirements  that were not key to the initial vehicle were grouped into three categories,  each more important than the next, but none important enough to be added  to the initial vehicle if not achievable within the resources available to the  developer. Product developers stated that several requirements would have  been difficult to achieve without additional time and money to develop  solutions. When they explained this to the customer and pointed out the  associated high risks for achieving the requirements within the time  frames, the customer was willing to defer these requirements to future  versions of the system in order to meet the planned development schedule.
The requirement for the Tactical Unmanned Aerial Vehicle’s imagery  capabilities—that is, the electro-optic and infrared sensor systems that are  used for observing targets—is a good example of this flexibility. The  customer originally wanted the imagery to have at least a 90-percent  probability of recognizing and identifying targets from an altitude of 6,000  to 8,000 feet. On the basis of a systems engineering analysis, the product  developer concluded that this was not achievable with the technology  available within the 1-½ year time frame and that more time and money  would be required to develop technology that would meet the capability. As  a result, the customer reduced the requirement to 70 percent, which was  both technically feasible and still useful to the customer, and made  90 percent a goal that might be achieved in future generations.
A similar combination—a cost limitation and urgent customer need—led to  flexible requirements on the Global Hawk unmanned aerial vehicle. A  development cost limit set by DOD executives forced the customer and the  program office to prioritize requirements into what could be achieved  within the limit. Additionally, the customer, the United States Joint Forces  Command, assessed the usefulness of the system and stated the basic  capability tested during the Advanced Concept Technology Demonstration  could provide utility today. This assessment increased the importance of  quick delivery and forced the requirement setters and the product  developer to look at an evolutionary approach to meeting their needs,  planning for a basic capability in the first product line and more advanced  capabilities added over time. Together, these constraints gave the program  manager leverage in persuading the user to reduce or defer requirements.  For example, the requirements manager wanted a more advanced imagery  system in the first product line. However, the program manager pointed out  to the requirements manager that the cost and time needed to include this  capability would exceed the cost limit. In response, the requirements  manager agreed to defer this capability to a later version of the product.  Figure 13 shows the Global Hawk unmanned aerial vehicle.
section_title: Less Successful Cases Maintained Inflexible Requirements Prior to Launch
paragraphs: On the weapon system programs that did not meet development objectives,  the products’ performance requirements were generally not flexible prior  to launch—a condition compounded by the fact that systems engineering  had not been done in time to reveal gaps between requirements and  resources. The requirements demanded advanced technologies that were  not yet proven or available. The matching of requirements with available  resources was not done until well into the product development program.  However, the requirements were not reduced. Instead, additional  resources, including technologies, time, and money, had to be added. We  have reported on the negative consequences of basing cost and schedule  estimates on the hoped-for success of such technologies. Without the  countervailing presence of systems engineering knowledge from the  product developer or the customer’s acceptance of an evolutionary product  development approach, flexibility was difficult to engender.
subsections: section_title: Army’s Comanche Helicopter
paragraphs: In the Comanche program, initial requirements for attacking targets and  performing other tasks, when combined with the desire for small size and  light weight, called for a unique capability compared with existing  helicopters. Meeting the size and weight requirements depended on new  technologies such as advanced forward looking infrared and integrated  avionics. Both size and weight were critical elements of a mission  equipment package that was supposed to reduce the pilot’s workload while  improving aircraft performance in the areas of communications, targeting  range, and pilot capabilities. The program office stated that at program  launch the technologies were still conceptual in nature, needing an  investment in time and funding before they would be mature enough to  satisfy requirements. As a result, Comanche was expected to provide a  quantum leap in product performance using new technologies that were  not fully understood at the time the program began.
Not only did the customer consider the requirements that made these  technologies necessary as not tradable, they were also confined by weight  and cost restrictions placed on the program. The cumulative effect of  several competing requirements—low cost, reduced weight, long range,  and increased lethality—resulted in solutions that relied on advanced  technologies. For example, a more mature target sensing technology that  might have met performance requirements was rejected because it weighed  too much. The Army decided to launch the program despite the significant  lack of knowledge about the needed technologies, leaving a mismatch  between requirements and available resources, and chose to develop the  new technologies during the product development program. This action  placed the burden of maturing technologies onto the program manager  during product development. Difficulties encountered maturing these  technologies to meet unyielding user requirements contributed, in part, to  the program’s significant cost and schedule increases.
Army’s Crusader Artillery Vehicle The Army’s Crusader program also had competing requirements that— when taken together—resulted in an inflexible system solution.  Requirements for firing range, accuracy, rate of fire, lethality, and resupply  remained inflexible prior to program launch. Similar to the Comanche  program, this inflexibility limited the technical solutions that could meet  the requirements and forced the program to be launched with immature  technologies. Many of these technologies—such as the automated  ammunition loading and handling system and the actively cooled cannon  barrel—were considered technological firsts for U.S. artillery systems. As a  result, the Crusader program is expected to take over 14 years and cost  over $4 billion to develop.
section_title: Army’s Brilliant Anti-armor Submunition
paragraphs: The BAT program also had inflexible requirements that exceeded available  resources at the start of the program. The BAT operational concept is based  on a need to locate targets from great distances using advanced sensor  technologies, which are then required to guide the weapon to closer ranges,  where the submunition would engage the target. The acoustic target  technology was the most important technology needed to meet the  weapon’s performance requirements, and the contractor proposed a  weapon concept based on it. The Army accepted the concept and drafted  performance requirements based on the acoustic technology. In this case,  there was limited flexibility to negotiate trade-offs between requirements  and resources given the small margin of error for a munition to hit the  target. Consequently, the key to matching requirements with resources  depended on committing the necessary resources to close gaps after the  program was launched.
At the start of product development, the cost and schedule targets for  BAT’s development were set without knowledge about the feasibility of the  performance requirements. While the requirements were based on a  specific technology, it was not mature enough for inclusion onto a product.  In fact, the technology was still being defined in paper studies. The Army  did not prototype this technology until almost 6 years after the program  was launched. During its product development program, the BAT  experienced significant development cost and schedule increases. Program  officials attribute these increases, at least in part, to unknowns about the  new technologies. Because requirements remained inflexible, the product  developer was forced to add resources to achieve a match between the  two.
section_title: Balance in the Roles of the Customer and Product Developer Makes for Effective Trade-offs
paragraphs: With knowledge gained from systems engineering and flexible  requirements as preconditions, successful product development programs  benefited from an environment in which product developers and customers  had balanced roles and shared responsibilities for setting product  requirements. In two successful cases, product managers were responsible  for both (1) translating customer wants into product requirements and (2) developing and delivering the products within available resources. This  dual responsibility allowed them to make reasonable trade-offs between  performance requirements and resources prior to committing to product  development. In two other successful cases, requirement setters and  product developers had equal roles in establishing and changing  requirements, using knowledge from systems engineering as a guide. In  unsuccessful cases, there was no parity in decision-making between the  requirement setter and the product developer; the requirement setter held  the upper hand by controlling the requirements without being bound by  cost and schedule considerations or the developer’s resources.
subsections: section_title: Successful Programs Benefited From Parity in Decision-making Between Customer and Developer
paragraphs: In the successful commercial cases we examined, the product developer  had ultimate responsibility for defining the product’s requirements but  worked intimately with the customers to understand their needs. The firms  established an environment that teamed customer representatives with  product development engineers in setting mutually agreeable product  requirements. Representatives from one commercial firm told us that they  feel responsible for understanding a customer’s operations better than the  customer does. They achieve this through their marketing people. Once the  marketing staff understands a customer’s wants, they can work with the  design and production engineers to develop a product solution that will  meet the customer’s needs and can be met with company resources. The  product manager makes final decisions on the product’s requirements, but  is mindful of the fact that a dissatisfied customer will not buy the product.
subsections: section_title: Caterpillar’s 797 Mining Truck
paragraphs: An example of how this parity in decision-making worked is the Caterpillar  797 mining truck engine. To meet the customers’ needs, the engine had to  produce at least 3,400 horsepower. For maintenance and serviceability  reasons, some mining companies wanted a single engine to produce this  power, which Caterpillar’s marketing staff translated into a requirement.  However, Caterpillar did not have such an engine and its systems  engineering analysis showed that a new single-engine design could not be  matured and demonstrated within the 18-month cycle time limit. Thus,  product developers proposed a twin-engine design that met the  horsepower requirement but used an existing engine. Caterpillar’s  customer representatives presented this proposal to the customers, as well  as their position that the firm could either deliver the truck powered by  twin engines in 18 months or take longer to deliver a single engine truck— but not both. Most customers accepted the proposal and the position.
section_title: Bombardier’s New Regional Jet
paragraphs: Similar parity in decision-making was evident concerning Bombardier’s  fly-by-wire technology. The airlines wanted fly-by-wire on the BRJ-X  aircraft because of the benefits. The benefits, which Bombardier’s systems  engineering confirmed included 96 percent fewer parts and 1,250 fewer  pounds than hydromechanical controls. These benefits would reduce  engine power requirements, saving about 3 percent of fuel capacity and  reducing the need for maintenance significantly. All of these savings  responded directly to the airlines’ performance needs. However, because  this technology had never been integrated onto a regional jet before,  Bombardier’s product development engineers believed it to be risky. Also,  since the workforce of regional jet pilots had never used fly-by-wire, they  would have to be trained on how to use it, and system engineers who had  the authority to accept or reject the technology would not commit to  including the technology unless it was demonstrated first. Bombardier  closed the technology gap by demonstrating it on an existing aircraft. At  that point, Bombardier was confident in including fly-by-wire in the  product requirements for the BRJ-X.
section_title: Less Successful Cases Did Not Have Parity Between the Requirement Setter and the Product Developer
paragraphs: In two less successful cases, the customer set the product requirements  with comparatively little input from the product developer. These programs  were initiated with cost and schedule estimates that were, for the most  part, based on the customer’s requirements and a notional design. The  product developer was not an equal partner in setting product  requirements and had to launch programs without requisite resources.
subsections: section_title: Navy’s Radio Frequency Countermeasures System
paragraphs: Before product development began, the program manager for the Radio  Frequency Countermeasures system had been working with the F/A-18 E/F  customers to match its needs with available resources. The program  manager told us that both parties believed they had a good match because  the technology was mature. Also, both parties agreed that an  approximately 5-½ year development cycle would deliver the desired  product. However, prior to program launch, as part of the normal process  used for approving product requirements, the proposed Navy requirements  were distributed to the other services. The Air Force became interested in  the program and requested that its needs for the F-15 and B-1 be combined  with the Navy’s to develop a common system. The Air Force’s requirements  were much more demanding than the Navy’s because more power was  needed to generate the electronic countermeasures for the F-15, a less  stealthy aircraft than the F/A-18 E/F. These additional needs created a  technology gap that required either more time or trade-offs.
The product developer analyzed the technology gaps and determined that,  given projected time and money, the effort required to meet them would  take close to 7 years. The Navy customer would not agree to the additional  time and insisted that the product developer deliver a product to meet the  expanded needs within the original 5-½ year time frame. The customers, in  sole control of requirements setting, thus firmed the product requirements.  The program manager informed us that he had little decision-making  authority in the matter and was compelled to accept the customers’  schedule. The result was an imbalance between product requirements and  resources, which also resulted in cost and schedule increases. Because of  the delays in developing the system, the Navy plans to incorporate a less  effective substitute onto the F/A-18 until the product is ready. These  aircraft will have to be retrofitted with the system when it becomes  available—a costly and inefficient process. Ultimately, the trade-offs that  were unpalatable before launch became unavoidable after launch.
section_title: Army’s Comanche Helicopter
paragraphs: Parity between the requirement setter and the product developer did not  exist on the Comanche program. As discussed previously, the Comanche  product development program included immature avionics technologies  that were needed to meet demanding and inflexible performance  requirements. These technologies represented significant gaps between  what the customer wanted and what the program office could confidently  deliver given the resources available. Prompted by concerns about the time  needed to advance the necessary technologies, the program manager  proposed trading off some requirements. However, requirements managers  informed us they were unwilling to accept trade-offs and that they believed  the program manager was too risk averse. Consequently, the program was  launched with the requirements intact, despite the program manager’s  concerns, and, cost and schedule estimates for its development have  doubled.
section_title: Characteristics of DOD’s Acquisition Process Make It Hard to Match Expectations and Resources Before Program Launch
paragraphs: Within DOD’s traditional acquisition process, it is difficult to employ  systems engineering in making trade-offs, maintain flexibility in customer  expectations, and put the product developers in a balanced position  relative to the customers. As a result, customer needs—however  legitimate—are translated into product requirements that make  unreasonable demands on available resources. This pattern is perpetuated  because resources are generally added later during product development to  close any gaps between requirements and resources. The mechanics of  obtaining funding and getting approval to start an acquisition program  dictate that events proceed in the following sequence: set requirements,  obtain funding, launch program, and perform systems engineering. This  sequence places the knowledge that is needed to identify resource gaps— and the power that it gives to the product developer—at a disadvantage to  shape requirements and improve program outcomes. Other aspects of the  process reward behaviors that can put pressure on requirements, making  them less flexible and more difficult to meet. For example, the services  must clearly differentiate a new weapon’s performance characteristics  from alternatives to successfully compete for funding, encouraging  detailed cost and schedule estimates to be made with little knowledge from  systems engineering. The weapon system programs that did match  requirements and resources before launch—the Tactical Unmanned Aerial  Vehicle and the Global Hawk—represented departures from this process  that benefited greatly from the intervention and protection of top-level  individuals.
DOD has recently changed the requirements setting and acquisition  processes to better reflect best commercial practices. Specifically, DOD  has revised its policy for approving new development programs by  providing more latitude for technology maturation and other knowledge-generating activities to take place before a program is  launched. The acquisition policy also supports the evolutionary  development approach as the preferred method of product development. In  addition, DOD has revised its policy for developing requirements, calling  for a staged or stepped approach that will dovetail with evolutionary  acquisitions. By itself, however, the policy can do little to foster the early  matching of customer wants with developer resources because the  mechanics of obtaining funding and the pressures on requirements setters  are unchanged. The incentives of the process, unlike policy, are more likely  to be seen in decisions made on the funding and approval of individual  programs.
subsections: section_title: Current Process Puts Requirements Setting and Systems Engineering on Opposite Sides of the Launch Decision
paragraphs: DOD’s acquisition process makes it difficult to know what resources will be  needed to meet requirements before launching a program. Within this  process, systems engineering does not take place until a program has been  launched and cost and schedule targets have been set. One reason for this  is that DOD does not typically sign contracts with product developers, who  are responsible for systems engineering, until after the requirements have  been written. Once an acquisition program is approved and product  development is funded, a product developer can begin systems engineering  and gaps between requirements and resources can be identified. However,  this process does not yield knowledge about the requirements’  achievability until well into product development. Even when requirements  are in draft at the time of launch, they are hard to change because user  representatives have already invested significant effort in preparing them.  This sequence of events—setting requirements, launching an acquisition  program, contracting with a product developer, and conducting systems  engineering—narrows the options for closing gaps primarily to increasing  cost and schedule estimates.
DOD’s process for setting requirements can begin many years before a  program is launched, and it can be several years after the program is  launched before systems engineering knowledge is obtained. The time  period from the start of the definition of the user’s needs to the arrival of  systems engineering knowledge that often uncovers gaps can approach 10 years in some instances. More than 30 organizations within the  requirements community may have a hand in determining a weapon  system’s performance requirements before a contractor with systems  engineering expertise can identify the gaps between requirements and  available resources. This process means the “doability” of the requirements  is often not known with certainty until well into product development or  until a significant percentage of funds planned to develop the system has  been invested. By this point in time, customers’ expectations have been set,  making it difficult to change requirements if gaps between requirements  and available resources are found.
The process used to set requirements and begin development on the Army’s  Crusader program provides an example that illustrates the mechanics of  the process (see fig. 14).
The user’s representative for the Crusader (the Army’s Training and  Doctrine Command artillery school at Fort Sill, Oklahoma) began drafting  the performance requirements for the Crusader in 1990. The process to  define the requirements took approximately 4 years. During this time, the  user representative framed the needed features of the Crusader and  conducted exhaustive analyses and trade studies to identify the optimal  point to set a specific requirement. Also, as required by DOD acquisition  policy, the user representative analyzed a notional Crusader that met these  requirements against other alternatives, such as improved versions of  existing artillery systems. The result of these analyses was a set of  requirements from the user representative and notional design prepared by  the Army program office. In 1994, the requirements for the Crusader system  were approved, the acquisition program was launched and funded, and  program cost and schedule targets were baselined.
By this time, key features of the Crusader were defined. For example, the  Crusader would have to use liquid propellant—a revolutionary  technology—to propel the projectiles far enough to meet the optimal range  requirements. Also, because the Crusader was required to stay on the firing  line during rearming and refueling, it would have to have an armored and  automated resupply vehicle to keep the resupply crews protected.
After the program was launched, the Army entered into a contract with  United Defense Limited Partnership to develop the Crusader. Over the next  4 years, the firm applied systems engineering practices to the requirements  in order to develop a preliminary design. During this process, the firm had  to make trade-offs in order to develop a design that could stay within  resources. By 1996—2 years into product development and 6 years since  the need for the Crusader was identified and determined—United Defense  Limited Partnership concluded that resources needed to develop a liquid  propellant technology were not available. The firm’s only option was to  accept a less capable technology—a conventional solid propellant,  increasing the risk that the Crusader’s firing range requirement may not be  achieved. According to Crusader officials, United Defense Limited  Partnership did not develop a good preliminary design of the system until  1998 or about 8 years after the user representative began the requirements  setting process.
section_title: Incentives of Current Process Create Pressure on Product Requirements
paragraphs: Several factors in the DOD environment create incentives to set  requirements high and to resist trade-offs. The competitive nature of the  process to justify a new program puts great pressure for a potential  weapon’s requirements to stand out. Unlike the commercial environment,  the user representatives are separate from both the customer and the  product developer, and thus play a unique role that has different interests.  Once user representatives get a set of requirements through the very  difficult and lengthy process of approval before a program can be  launched, they are understandably reluctant to change them. Finally, the  DOD customer—unlike its commercial counterpart—is more tolerant of  schedule delays and cost increases.
subsections: section_title: Pressures on the Requirements-setting Process
paragraphs: The competition within DOD to win funding and get approval to start a new  program is intense; establishing the basic need and writing requirements is  perhaps the most important step in the process. This creates strong  incentives for requirements setters to write performance requirements that  will make their particular weapon system stand out from existing or  alternative systems. Much of the requirement-setting activity prior to  initiating a program is devoted to proving the superior cost-effectiveness of  the preferred system over others, with less consideration given to the  resources that will be needed to develop the system. If user representatives  do not write requirements that will withstand scrutiny and prevail over  alternatives, the program could be killed and no capability is gained for the  customer. Costly, labor-intensive studies, such as analyses of alternatives,  are done to determine the best possible solution to meet the user’s needs  and to beat the competition. At the same time, overall DOD funding  constraints put a high priority on affordability, making it important for  program sponsors to provide cost estimates that will fit within the funding  constraints. Instead of forcing trade-offs, challenging performance  requirements, when coupled with other constraints, such as cost or light  weight, can drive product developers to pursue exotic solutions and  technologies that, in theory, could do it all.
To simplify requirements and foster trade-offs, DOD has adopted the  practice of identifying only a limited number of key performance  parameters—“must-haves” for the customer—for each weapon system.  This practice may help to prioritize requirements. However, if the  parameters are not informed by systems engineering, given the pressures  inherent in the process, they still result in unrealistic demands made of  existing resources. For example, the Army’s Crusader program had only  five key performance parameters, including characteristics such as  rate-of-fire, range, and speed, but they were dependent on about 500 other  subordinate performance requirements that were defined before systems  engineering was done by the product developer.
The unique role of the requirement setter in DOD’s acquisition process can  also put pressure on requirements. DOD’s requirement setters are outside  of the acquisition community and often represent an operational function  such as air combat, artillery, or armor. They are not the actual user like an  operating air wing or Army brigade, but serve as a representative of the  user. Unlike the commercial world, where a customer’s wants are  represented by someone within the product developer’s organization,  DOD’s requirement setters are, for the most part, separate and independent  of the customer, program manager, and the product developer. A user  representative can work directly with DOD science and technology  organizations and defense firms to obtain information about new  technology, and thus may be less willing to defer to a program manager’s  advice on these issues. Also, while a customer is focused on current  problems and near-term performance, a user representative tries to look at  longer term needs. In fact, the fear that DOD may not procure another such  system for several years creates incentives for user representatives to  reach for the most capability possible because they do not know when or if  they will get another opportunity to develop and acquire the next weapon  system.
The Comanche helicopter program is a good example of these pressures at  work. The Comanche was initiated in the early 1980s as a family of  lightweight, multipurpose helicopters whose operation and support costs  would be 50 percent less than the Vietnam-era fleet. The program was  originally expected to offer as good a technical performance as possible  within clearly stated—and low—unit cost goals. The requirements were  simple at the start. However, once the requirement setting process began,  the program emerged as it is today—a threat-based program to yield the  next-generation, high-performance helicopter—at a cost significantly  higher than the existing Apache. It was justified as being faster, stealthier  and smaller than the Apache helicopter. Resource gaps were identified  after the program was launched, when the product developer began to  obtain knowledge through systems engineering and technology  development about what it would take to meet the requirements.
Leadership in the Army has called for a transformation in Army forces and  equipment to a lighter, more mobile force that can be deployed more  quickly and easily. The impact this change is having on Crusader’s  requirements further illustrates how DOD incentives can drive  requirements and ultimately impact program outcomes. The preliminary  Crusader design —which met all key requirements—was considered too  heavy for the new, lighter force. In fact, the Army Chief of Staff said he  would like to have the Crusader’s weight reduced to the point that twice as  many vehicles could be moved with the same amount of transport aircraft.  With the program’s future at stake, the user representative, program  manager, and product developer are working together to reduce the overall  design weight of the system from 55 to 38 tons. Because of this new  priority, previously untraded requirements—such as degree of crew  protection, time on the firing line, and the need for tracked vehicles—are  now being examined to see if they can be reduced to save weight.
A specific Crusader requirement is illustrative. In developing the original  set of requirements, the user representatives determined that the  Crusader’s effectiveness would be maximized if the Crusader could avoid  having to leave its battle station to reload and refuel. To meet this  requirement, the resupply vehicle would have to reload and refuel the  howitzer at the battle station. This requirement necessitated that the  resupply vehicle be armored, tracked, and fully automated so that the crew  would not be exposed to enemy fire. This not only made for a more  effective howitzer but also distinguished the Crusader from other  candidates. In looking for weight reductions, the Army is considering  pulling at least some of the howitzers off station to reload and refuel. If it  relaxes that requirement, the resupply vehicle and crew can do its work  under safer conditions. In turn, the vehicle would not have to be as heavily  armored, automated, or tracked. In fact, the Army is considering using  existing trucks, which are far lighter and less expensive than a new  resupply vehicle. In our view, the changed circumstances of the Crusader  program—past the approval gate, a product developer deep into systems  engineering, and a top level mandate to reduce weight—created incentives  to make requirements trade-offs that were not acceptable before.
section_title: Calcification and Customer Acceptance Do Not Encourage Trade-offs
paragraphs: Once established, system requirements undergo incredible scrutiny and  review by a myriad of interests within DOD’s process. For example, the  Crusader draft requirements were circulated to approximately 30 organizations throughout the Army, other services, operational  commands, and the development community for review. This process  yielded 943 comments. To incorporate or otherwise dispose of each  comment, a joint working group with representatives from all of the  reviewing organizations was established. The group incorporated 702 of  the 943 comments into the requirements, the cumulative effect of which  was to add, rather than to trade, requirements. As one official stated “it is  generally not the practice to reduce or eliminate requirements but to add  more to appease a particular party.” This lengthy and cumbersome review  process tends to calcify weapon system requirements before product  development begins and knowledge from systems engineering can be  obtained.
The practice of breaching cost and schedule objectives to meet difficult  requirements would not persist without a customer’s cooperation. Unlike  commercial customers, DOD customers tend to be tolerant of cost  overruns and delays in order to get a high-performance weapon system.  Traditionally, customers have been willing to wait long periods of time for a  highly desirable system that they feel will provide them the longest lasting  capability. They would rather wait for the most desirable system to be  developed than accept a less capable system, thinking that they may not get  the opportunity to acquire a new or modified system in the future. Again,  the Comanche program provides insight. At the time the program was  started, the Army expected to receive the first operational helicopter in  1996. The development program has encountered delays; some related to  weight, cost, and performance requirements that demanded immature  technologies. However, the Army has chosen to keep the requirements  generally intact and to do without the helicopter instead of fielding a less  capable system more quickly. The Army now expects to have an initial  operational capability in 2006, 10 years after the initial target date.
section_title: More Successful Weapon System Programs Have Departed From the Normal Process
paragraphs: The two DOD programs we examined—Tactical Unmanned Aerial Vehicle  and Global Hawk—that set requirements in a way that approximates best  commercial practices were departures from DOD’s normal process.  Specifically, (1) their origins as Advanced Concept Technology  Demonstrations enabled them to hire a product developer much earlier  than is traditionally the case and (2) they benefited greatly from the  personal intervention of service and DOD executives who created and  enforced conditions that were conducive to making trade-offs between  wants and resources. Other new programs, such as the Joint Strike Fighter  program, showed that traditional incentives to speed a program along still  existed and could increase risk in product development.
The Advanced Concept Technology Demonstrations, for the Tactical  Unmanned Aerial Vehicle and the Global Hawk, showed the feasibility of  developing a system that could do what the customer wanted before a  commitment to product development was made. The sequence of events  leading to program launch on each of these programs was similar to those  of successful commercial firms. The demonstrations involved the  customer, user representative, program manager, and product developer in  a more integrated fashion than is normally the case. Moreover, by building  prototypes for the demonstrations, the product developers had to conduct  a significant amount of systems engineering before the programs were  launched. Consequently, the demonstrations provided valuable knowledge  that allowed the customer, user representative, and product manager to  define a set of product requirements that could be met within resources.  Most importantly, the demonstrations were conducted before program  launch, before cost and schedule targets were set.
Both programs had unusual intervention by top-level individuals that set  resource constraints and encouraged evolutionary acquisition strategies.  On the Tactical Unmanned Aerial Vehicle program, the top military  acquisition executive personally met with the head of the user  representative’s organization and struck an agreement that the product was  to be fielded in stages, with the first stage being a very basic system. This  agreement had the effect of establishing mutual interest in the same  program outcome and shielded the program from criticism by either  community. The personal involvement of the Under Secretary of Defense  for Acquisition, Technology, and Logistics helped set the stage for Global  Hawk’s evolutionary approach to meeting requirements. The Under  Secretary insisted on an initial capability that could be developed within a  fixed budget while providing the flexibility to defer other requirements to  succeeding versions. In both cases, this top-level intervention allowed  requirements to be flexible and gave the product developers parity with the  requirements setters in influencing requirements. Equally important, we  believe the intervention signaled support for the programs, which eased  some of the pressures that normally accompany efforts to get programs  approved. In each case, while there was a potential for success, it came not  so much from a well-established process but from exceptional behavior  from senior-level leaders. If, for some reason, leadership or priorities  change, the process may not ensure success.
section_title: Constructive Changes in DOD’s Policy Not Enough to Match Customer Expectations With Developers’ Resources
paragraphs: While DOD has taken steps that could help it more efficiently acquire its  weapons, these steps have not substantively changed the mechanics or  incentives of the requirements setting process. While the revised  acquisition policy provides opportunities for improvement, it is not specific  about when to match wants and resources. In fact, the sequence of  events—setting requirements, obtaining funding, launching an acquisition  program, and conducting systems engineering—remains essentially the  same as before the revision. Recent experiences with the Global Hawk and  the Joint Strike Fighter programs indicate that traditional pressures on the  requirements process are still strong.
subsections: section_title: Recent DOD Policy Revisions Are Supportive of Best Practices
paragraphs: DOD has recently revised its policies for operation of the defense  acquisition and requirements setting processes. The acquisition policy  reflects best commercial practices and emphasizes better use of  evolutionary acquisitions, more reliance on mature technology, and  reduced cycle times and costs. It recognizes shorter acquisition cycle times  as critical to making the best use of advanced technologies and  evolutionary requirements as a way to reduce cycle times. The Under  Secretary of Defense for Acquisition, Technology, and Logistics issued  related guidance stating that DOD’s objective will be to achieve acquisition  cycle times—from program start to initial fielding—no longer than 5 to 7  years. The revised acquisition policy also states a clear preference for  evolutionary acquisitions over “single step to full capability” acquisitions.  DOD also revised its acquisition process, which is now four phases:  (1) concept and technology development, (2) system development and  demonstration, (3) production and deployment, and (4) operations and  support (see fig. 15). In addition, guidance that directs the requirements  setting process was revised to include an emphasis on the use of  time-phased performance requirements in support of evolutionary  acquisitions.
The decision to launch a program would normally take place between the  first and second phases. The revised policy states that the decision is  dependent on three factors: technology maturity, validated requirements,  and funding. Assuming that technology is found to be mature, at the start of  the system demonstration and development phase, the policy requires the  weapon’s requirements to be formally approved and full funding to be  provided for the remainder of the program. This is the point at which a  program manager would commit to performance, cost, and schedule  estimates and DOD would commit to provide that level of resources. DOD  later would award contracts to product developers to conduct the majority  of systems engineering necessary for designing and building the product. In  that regard, little has changed under this revised approach. It is still  difficult to know what resources will be needed to meet a set of  requirements before making the decision to launch the program.
section_title: Pressures on Requirements Are Still Powerful
paragraphs: While DOD policy encourages evolutionary acquisition approaches, it will  not be successful if traditional incentives for setting high and inflexible  requirements persist. Recent developments on the Global Hawk program  indicate the continuing pressure programs face to increase requirements.  The initial Global Hawk system provides a capability for high altitude  reconnaissance that the customer has stated is acceptable for meeting its  short-term needs. Future generations would add capabilities to match the  U-2—a manned reconnaissance airplane—as technology, money, and time  become available. The Air Force had embarked on an evolutionary  acquisition that defines the first generation of the Global Hawk based on  the system already demonstrated with performance, supportability, and  producibility enhancements. However, the Air Combat Command, the user  representative, indicated that it would accept nothing short of the same  capability found in the current U-2, despite the customer’s views. The  Command did not have confidence that the evolutionary approach would  receive the support and resources to attain the U-2’s capabilities in future  generations. As a result of pressure from the user representative, the Air  Force proposed a revised acquisition strategy that while still evolutionary  in nature, accelerated the inclusion of advanced technologies onto the  aircraft to provide U-2 capabilities sooner. In December 2000, the Deputy  Secretary of Defense elected not to approve the revised acquisition strategy  at this time. While we did not review the revised approach in detail, it did  appear risk had been added to the program as requirements increased and  resources changed.
The Joint Strike Fighter program is another example in which traditional  incentives can result in decisions contrary to best practices. While DOD  has designated the Joint Strike Fighter as a flagship program for acquisition  reform, it is now 4 years into its acquisition program and has not yet  achieved a match between requirements and resources—particularly in the  form of the technologies needed. By best practice standards, none of the  fighter’s critical technology areas that the program office has identified are  expected to be at readiness levels acceptable for entry into the engineering  and manufacturing development phase, which is scheduled for 2001.  Reminiscent of the Comanche, numerous conflicting demands that the  fighter achieve high performance and low costs have resulted in  requirements that must be satisfied with several technical advances. The  requirements have also proven to be relatively inflexible. As we reported  in May 2000, delaying this phase of the program until technologies are  mature would improve the chances that the Joint Strike Fighter will be  fielded as planned. However, despite not having the requisite knowledge  for the technologies, DOD has deemed the risks manageable and proposes  to proceed with the program as planned. Such a decision reinforces  traditional incentives and increases the likelihood for future problems.
section_title: Conclusions and Recommendations for Executive Action
subsections: section_title: Conclusions
paragraphs: Managing the setting of a product’s requirements in a way that matches the  customer’s needs with the developer’s resources is critical for a successful  product development. This kind of success—delivering weapons that meet  needs within predicted costs and time frames—is essential if DOD is to get  what it wants from its huge modernization investment. The best practices  for balancing needs and resources before committing to a new product or  weapon system development are to (1) conduct systems engineering to  illuminate what has to be done to match the wants with the resources;  (2) establish fixed cycle times for an initial product within an overall  evolutionary approach to foster flexibility needed to make key trade-offs;  and (3) maintain parity between requirement setters and product  developers when translating customer needs into product requirements.  These practices provide for identifying gaps between wants and resources  as well as solutions—whether by trading off needs or investing more  resources—before setting requirements and starting development.
The environment for commercial products provides incentives that  encourage these practices. Like DOD, commercial customers have an  initial advantage in that they have high expectations of a new product, they  do not have unlimited time and money, and ultimately, their needs have to  be met. While firms are motivated to offer a customer a product that meets  these needs, they are keenly aware that promising more than they can  deliver will ultimately disappoint the customer. Consequently, leading firms  consciously attempt to manage customer expectations and put themselves  in a good position to negotiate a reasonable set of product requirements.  They do this in several ways. First, because they are investing their own  resources to develop the product, they are at liberty to conduct systems  engineering early—an investment they consider small relative to the overall  cost of developing a product. Second, they have a proven track record for  delivering products on time and making promised improvements to  succeeding versions of the product. Third, they play a prominent role in  setting and agreeing to product requirements. Together, these factors help  create the credibility, confidence, and trust that are essential to maintaining  the flexibility needed to match their resources with the customer’s needs  before launching a new product development.
Within DOD’s traditional acquisition process it is difficult to gain  knowledge and maintain flexibility in requirements prior to committing  resources to an acquisition program. As a result, customer needs are  translated into product requirements that often make unreasonable  demands on available resources. There are several reasons for setting  weapon system requirements this way. Some are the following:   The mechanics of the program approval and funding process force a  sequence of events that keeps a product developer relatively uninvolved  in shaping requirements, compared with best practices, causing product  development to begin with cost and schedule targets untempered by  systems engineering knowledge.
Unique or demanding requirements can help gain approval to fund a  new program because aiming too low can result in losing out to other  programs, thus denying any new capabilities.
There is some mistrust in an evolutionary approach, specifically  regarding whether future improvements will be approved; consequently,  attempting to reach for the full capability in a single step can be seen as  a safer course of action.
The challenging and long process to get requirements approved hardens  requirements setters against trade-offs—in fact, the process of getting a  requirement approved can actually encourage additions to garner  support.
Unlike the commercial world, DOD customers have proven to be  tolerant of cost and schedule increases once a program is underway.
Seen in this light, while the pattern of weapon system requirements  outstripping planned resources is inefficient, it is not irrational; within the  DOD environment, this approach is successful in starting programs that  eventually provide superior capabilities. This approach also has negative  consequences. First, the additional time and money that must be invested  after launch still yield the same capability called for by the original  requirements, effectively lowering the buying power of the investment.  Second, the unplanned nature of the additional investment generally  requires weapon system quantities to be lowered or money to be taken  from other programs. Third, when a requirements setter is unwilling to  make trade-offs before launch but later elects to accept delivery delays to  meet those requirements, the implicit trade-off is to have the customer do  without any of the new capability for a longer period of time. The challenge  in adopting best practices, therefore, is to make them rational—that is,  critical for success—in the DOD environment. Meeting this challenge will  take changes in both the mechanics and the incentives of the requirements  setting and program approval processes.
DOD’s recently revised policies could make it possible for a better  matching of customer needs and developer resources before program  launch. While these policies are sound and merit support, they retain the  mechanics of the old policies. Namely, requirements must be set before a  program can be approved and a program must be approved before the  money can be obtained to pay the product developer to conduct systems  engineering. Such mechanics deny the knowledge needed to match wants  with resources before starting a program. Similarly, the new policies must  overcome the still extant pressures brought to bear on requirements setters  to aim high and become inflexible. It took unique circumstances—starting  as Advanced Concept Technology Demonstrations and enjoying the  personal intervention of top DOD executives—to create the incentives on  the Global Hawk and Tactical Unmanned Aerial Vehicle programs for  making the trade-offs needed to match needs with resources. Even so, the  Global Hawk is struggling with pressures to reverse some of the trade-offs  and raise requirements. Other programs, such as the Joint Strike Fighter,  have requirements that outstrip resources, despite efforts to keep  requirements flexible and to treat the price of the aircraft as a requirement.
section_title: Recommendations for Executive Action
paragraphs: To realign the mechanics of the requirements setting and program approval  processes to bring more knowledge into the process of setting  requirements, we recommend that the Secretary of Defense require that the  systems engineering needed to evaluate the sufficiency of available  resources—knowledge, time, money, and capacity—be conducted in time  to help identify and make the critical trade-offs that precede the  formalization of requirements. One option is to allow the award of well-defined systems engineering contracts to prospective product  developers—contractors—before the system development and  demonstration phase.
To realign the incentives of the requirements setting and program approval  processes with the need to match available resources, we recommend that  the Secretary of Defense:  Reduce the pressures put on user representatives to set requirements  high to win the competition for program approval. One way to reduce  these pressures, drawing on the experiences of the Tactical Unmanned  Aerial Vehicle and Global Hawk, is to have higher level officials in the  services and DOD decide on the type of weapon system that is needed to  meet a valid need before the requirements setters begin detailed work  on framing a specific solution. Making such a decision earlier in the  process would ease the pressure on to set overly demanding and  inflexible requirements that will crush alternatives and win program  approval.
Require, as a condition for starting the system development and  demonstration phase for a weapon system—program launch—that  sufficient evidence exists to show there is a match between a weapon  system’s requirements and the resources the program manager has to  develop that weapon. Based on our current and past work on the best  practices of leading commercial firms, there is a key tool the Secretary  can use to define what resources DOD is willing to apply—establishing  limits on the time it takes to complete system development, such as not  to exceed 5 years. Further, having a formal agreement among the  requirements setters, program managers, and resource providers on  development and delivery of the required product would emulate the  best practice of establishing accountability for subsequent actions that  stray from the agreement.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD concurred with a draft of this report and its recommendations and  agreed that the requirements process needs to be better informed by  systems engineering in order to allow for the timely leveling of user needs  and developer solutions. It noted that the recently revised acquisition  policy takes the first steps in this new direction with guidance for  evolutionary acquisitions and the identification of knowledge points in the  acquisition process, but agreed that more progress needs to be made.  Specifically, DOD recognized that more knowledge in the setting of  requirements and the potential ability of the producer to meet those  requirements would provide a greater understanding of the time, cost, and  potential success of a program. DOD also recognized that the systems  development and demonstration phase can begin with a higher level  definition of requirements (that is, a higher level than the Operational  Requirements Document), which can be used by the requirements-setter  and the program manager, after some systems engineering work is done, to  facilitate trade-offs earlier and relieve some of the pressure on Operational  Requirements Document. Finally, DOD stated that the Acquisition Program  Baseline should require signatures from both the user and the resource  sponsor prior to program initiation signifying agreement that there is  sufficient evidence—such as determined by systems engineering,  demonstration of technology maturity, and adequacy of funding—that a  match between requirements and resources has been made.