Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes
Published: 2002-07-15T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: This report examines how best practices offer improvements to the way the Department of Defense (DOD) develops new weapons systems, primarily the design and manufacturing aspects of the acquisition process. Knowledge about a product's design and producibility facilitates informed decisions about whether to significantly increase investments and reduces the risk of costly design changes later in the program. Leading commercial companies employ practices to capture design and manufacturing knowledge in time to make key decisions during product development. First, the companies kept the degree of the design challenge manageable before starting a new product development program by using an evolutionary approach. Second, the companies captured design and manufacturing knowledge before the two critical decision points in product development: when the design was demonstrated to be stable--the second knowledge point--and when the product was demonstrated to be producible at an affordable cost--the third knowledge point. DOD has made changes to its acquisition policy in an attempt to improve its framework for developing weapons systems, but the policy does not require the capture of design or manufacturing knowledge or sufficient criteria to enter the system demonstration and production phases. In addition, it does not require a decision review to enter the demonstration phase of product development.

section_title: Letter
paragraphs: As you requested, this report examines how best practices offer improvements to the way the  Department of Defense develops new weapon systems, primarily the design and manufacturing  aspects of the acquisition process. It examines the attainment of design and manufacturing  knowledge and its use at critical junctures to make decisions about weapon systems’ readiness to  move forward in the acquisition process. We make recommendations to the Secretary of Defense for  improvements to weapon system acquisition policy to better align design and manufacturing activities  with best practices that have shown that the capture and use of key knowledge can result in better  cost, schedule, and performance outcomes.
We are sending copies of this report to the Secretary of Defense; the Secretary of the Army; the  Secretary of the Navy; the Secretary of the Air Force; the Director of the Office of Management and  Budget; the Director, Missile Defense Agency; and interested congressional committees. We will also  make copies available to others upon request. In addition, the report will be available at no charge on  the GAO Web site at http://www.gao.gov.
If you have any questions regarding this report, please call me at (202) 512-4841. Other contacts are  listed in appendix II.
subsections: section_title: Introduction
paragraphs: The Department of Defense (DOD) spends close to $100 billion annually to  research, develop, and acquire weapon systems, and this investment is  expected to grow substantially. Over the next 5 years, starting in fiscal year  2003, DOD’s request for weapon system development and acquisition funds  is estimated to be $700 billion (see fig. 1).
How effectively DOD manages these funds will determine whether it  receives a good return on its investment. Our reviews over the past 20 years  have consistently found that DOD’s weapon system acquisitions take much  longer and cost much more than originally anticipated, causing disruptions  to the department’s overall investment strategy and significantly reducing  its buying power. Because such disruptions can limit DOD’s ability to  effectively execute war-fighting operations, it is critical to find better ways  of doing business.
In view of the importance of DOD’s investment in weapon systems, we  have undertaken an extensive body of work that examines DOD’s  acquisition issues from a different, more cross-cutting perspective—one  that draws lessons learned from the best commercial product development  efforts to see if they apply to weapon system acquisitions. This report looks  at the core of the acquisition process, specifically product development  and ways to successfully design and manufacture the product. Our  previous reports looked at such issues as how companies matched  customer needs and resources, tested products, assured quality, and  managed suppliers and are listed in related GAO products at the end of the  report.
subsections: section_title: Best Practices of Leading Commercial Companies
paragraphs: Leading commercial companies expect their program managers to deliver  high-quality products on time and within budget. Doing otherwise could  result in the customer walking away. Thus, the companies have created an  environment and adopted practices that put their program managers in a  good position to succeed in meeting these expectations. Collectively, these  practices ensure that a high level of knowledge exists about critical facets  of the product at key junctures during development. Such a knowledge- based process enables decision makers to be reasonably certain about  critical facets of the product under development when they need this  knowledge.
To ensure the right level of knowledge at each key decision point in  product development, leading commercial companies separate technology  from product development and take steps to ensure the product design is  stabilized early so product performance and producibility can be  demonstrated before production. The process followed by leading  companies, illustrated in figure 2, can be broken down into the following  three knowledge points.
Knowledge point 1 occurs when a match is made between the  customer’s needs and the available resources—technology, design, time,  and funding. To achieve this match, technologies needed to meet  essential product requirements must be demonstrated to work in their  intended environment. In addition, the product developer must  complete a preliminary product design using systems engineering to  balance customer desires with available resources.
Knowledge point 2 occurs when the product’s design demonstrates its  ability to meet performance requirements. Program officials are  confident that the design is stable and will perform acceptably when at  least 90 percent of engineering drawings are complete. Engineering  drawings reflect the results of testing and simulation and describe how  the product should be built.
Knowledge point 3 occurs when the product can be manufactured  within cost, schedule, and quality targets and is reliable. An important  indicator of this is when critical manufacturing processes are in control  and consistently producing items within quality standards and  tolerances. Another indicator is when a product’s reliability is  demonstrated through iterative testing that identifies and corrects  design problems.
This report focuses on best practices for achieving knowledge points 2 and  3, particularly at how successful companies design and manufacture a  product within established cost, schedule, and quality targets. The  concepts discussed build on our previous reports, which looked at the  earlier phases of an acquisition, including matching customer needs and  available resources.
A key success factor evident in all our work is the ability to obtain the right  knowledge at the right time and to build knowledge to the point that  decision makers can make informed decisions about moving ahead to the  next phase. Programs that do this typically have successful cost and  schedule outcomes. Programs that do not typically encounter problems  that eventually cascade and become magnified through the product  development and production phases. As shown in figure 3, the effects of  not following a knowledge-based process can be debilitating.
section_title: DOD’s Traditional Approach to Product Development
paragraphs: DOD has historically developed new weapon systems in a highly  concurrent environment that usually forces acquisition programs to  manage technology, design, and manufacturing risk at the same time. This  environment has made it difficult for either DOD or congressional decision  makers to make informed decisions because appropriate knowledge has  not been available at key decision points in product development. DOD’s  common practice for managing this environment has been to create  aggressive risk reduction efforts in its programs. Cost reduction initiatives  that typically arise after a program is experiencing problems are common  tools used to manage these risks. Figure 4 shows the overlapping and  concurrent approach that DOD uses to develop its weapon systems. This  figure shows that DOD continues to capture technology, design, and  manufacturing knowledge long after a program passes through each of the  three knowledge points when this knowledge should have been available  for program decisions.
More important, the problems created by this concurrent approach on  individual programs can profoundly affect DOD’s overall modernization  plans. It is difficult to prioritize and allocate limited budgets among needed  requirements when acquisition programs’ cost and schedule are always in  question. Programs that are managed without the knowledge-based  process are more likely to have surprises in the form of cost and schedule  increases that are accommodated by disrupting the funding of other  programs. Because of these disruptions, decision makers are not able to  focus on a balanced investment strategy.
section_title: DOD’s Adoption of Best Practices
paragraphs: DOD is taking steps to change the culture of the acquisition community  with actions aimed at reducing product development cycle times and  improving the predictability of cost and schedule outcomes. DOD recently  made constructive changes to its acquisition policy that embrace best  practices. These changes focused primarily on (1) ensuring technologies  are demonstrated to a high level of maturity before beginning a weapon  system program and (2) taking an evolutionary, or phased, approach to  developing new weapon systems. Because these changes occurred in 2000  and 2001, it is too early to determine how effectively they will be put into  practice. While these are good first steps, further use of best practices in  product development would provide a greater opportunity to improve  weapon system cost and schedule outcomes.
section_title: Objectives, Scope, and Methodology
paragraphs: Our overall objective was to determine whether best practices offer  methods to improve the way DOD ensures that the design is stable early in  the development process and whether having manufacturing processes in  control before production results in better cost, schedule, and quality  outcomes in DOD major acquisition programs. Specifically, we identified  best practices that have led to more successful product development and  production outcomes, compared the best practices to those used in DOD  programs, and analyzed current weapon system acquisition guidance for  applicability of best practices.
To determine the best practices for ensuring product design and  manufacturing maturity from the commercial sector, we conducted general  literature searches. On the basis of our literature searches and discussions  with experts, we identified a number of commercial companies as having  innovative development processes and practices that resulted in successful  product development. We visited the following commercial companies:  Caterpillar designs and manufactures construction and mining  equipment, diesel and natural gas engines, and industrial gas turbines. In  2001, it reported sales and revenues totaling $20.45 billion. We visited its  offices in Peoria, Illinois.
Cummins Inc. (Engine Business group) designs and manufactures diesel  and natural gas engines ranging in size from 60 to 3,500 horsepower for  mining, construction, agriculture, rail, oil and gas, heavy and medium- duty trucks, buses, and motor homes. In 2001, the Engine Business  Group reported sales of $3.1 billion. We visited its offices in Columbus,  Indiana.
General Electric Aircraft Engines designs and manufactures jet engines  for civil and military aircraft and gas turbines, derived from its  successful jet engine programs, for marine and industrial applications.
In 2001, it reported earnings totaling $11.4 billion. We visited its offices  in Evendale, Ohio.
Hewlett Packard designs and manufactures computing systems and  imaging and printing systems for individual and business use. In 2001, it  reported revenues totaling $45.2 billion. We visited its offices involved in  the design and manufacturing of complex ink jet imaging equipment in  Corvallis, Oregon.
Xerox Corporation designs and manufactures office equipment,  including color and black and white printers, digital presses,  multifunction devices, and digital copiers designed for offices and  production-printing environments. In 2001, it reported revenues totaling  $16.5 billion. We visited its offices in Rochester, New York.
At each of the five companies, we conducted structured interviews with  representatives to gather uniform and consistent information about each  company’s new product development processes and best practices. During  meetings with these representatives, we obtained a detailed description of  the processes and practices they believed necessary and vital to mature a  product design and get manufacturing processes under control. We met  with design engineers, program managers, manufacturing and quality  engineers, and developers of the knowledge-based processes and policies.
During the past 5 years, we have gathered information on product  development practices from such companies as 3M, Boeing Commercial  Airplane Group, Chrysler Corporation, Bombardier Aerospace, Ford Motor  Company, Hughes Space and Communications, and Motorola Corporation.  This information enabled us to develop an overall model to describe the  general approach leading commercial companies take to develop new  products.
Our report highlights several best practices in product development based  on our fieldwork. As such, they are not intended to describe all practices or  suggest that commercial companies are without flaws. Representatives  from the commercial companies visited told us that the development of  their best practices has evolved over many years and that the practices  continue to be improved based on lessons learned and new ideas and  information. They admit that the application and use of these have not  always been consistent or without error. However, they strongly suggested  that the probability of success in developing new products is greatly  enhanced by the use of these practices. Further, because of the sensitivity  to how data that would show the actual outcomes of new product  development efforts might affect their competitive standing, we did not  obtain specific cost, schedule, and performance data. Most examples  provided by these companies were anecdotal. However, the continued  success of these companies over time in a competitive marketplace  indicated that their practices were important and key to their operations.  Furthermore, based on our observations during meetings at these  companies, it was apparent that because of the level of detailed process  tools developed for their managers and executive leadership these best  practices were a centerpiece of their operations.
Next, we compared and contrasted the best practices with product  development practices used in five DOD major acquisition programs.  Below is a brief description of each program we examined:  The F-22 fighter aircraft program. This aircraft is designed with  advanced features to allow it to be less detectable to adversaries,  capable of high speeds for long ranges, and able to provide the pilot with  improved awareness of the surrounding situation through the use of  integrated avionics. The F-22 program began in 1986 and entered limited  production in 2001. The Air Force expects to buy 341 at a total  acquisition cost (development and procurement) estimated at $69.7  billion.
The Patriot Advanced Capability (PAC-3) missile program. This program  is intended to enhance the Patriot system, an air-defense, guided missile  system. PAC-3 is designed to enhance the Patriot radar’s ability to detect  and identify targets, increase system computer capabilities, improve  communications, increase the number of missiles in each launcher, and  incorporate a new “hit-to-kill” missile. The “hit-to-kill” missile  capabilities represent a major part of the development program, as these  are not capabilities included in prior versions of the Patriot system. The  missile program began in 1994 and entered limited production in 1999.  The Army plans to buy 1,159 missiles at a total acquisition cost  estimated at $8.5 billion.
The Advanced Threat Infrared Countermeasures/Common Missile  Warning System (ATIRCM/CMWS) program. ATIRCM/CMWS is a  defensive countermeasure system for protection against infrared guided  missiles. The common missile warning system detects missiles in flight,  and the advanced threat infrared countermeasure defeats the missile  with the use of a laser. The combined system is designed for helicopter  aircraft. The common missile warning system is also designed for  tactical aircraft such as fighters. The program began in 1995 and is  expected to start limited production in 2002. The Army and the Special  Operations Command plan to buy 1,078 systems at a total acquisition  cost estimated at $2.9 billion.
The AIM-9X missile program. AIM-9X is an infrared, short range, air-to- air missile carried by Navy and Air Force fighter aircraft. The AIM-9X is  an extensive upgrade of the AIM-9M. The AIM-9X is planned to have  increased resistance to countermeasures and improved target  acquisition capability. A key feature is that it will have the ability to  acquire, track, and fire on targets over a wider area than the AIM-9M.  The AIM-9X program began in 1994 and entered limited production in  2000. DOD plans to buy 10,142 missiles at a total acquisition cost  estimated at $3 billion.
The F/A-18 E/F fighter aircraft program. This aircraft is intended to  complement and eventually replace the current F/A-18 C/D aircraft and  perform Navy fighter escort, strike, fleet air defense, and close air  support missions. It is the second major model upgrade since the F/A-18  inception. The development program began in 1992. The program  entered limited production in 1997 and full rate production in 2000. The  Navy plans to buy 548 aircraft at a total acquisition cost estimated at  $48.8 billion.
We selected these programs for review based on cost, schedule, and  performance data presented in the Selected Acquisition Reports for each  program. We also selected these programs because we considered them to  be in two basic categories—successful and unsuccessful cost and schedule  performance outcomes. This basis for selection was to compare and  contrast the development practices used on each with best practices used  by the commercial companies. For each program, we interviewed key  managers and design and manufacturing engineering representatives. In  some cases, we discussed design and manufacturing issues with  representatives of the primary contractor for the specific program to obtain  information on the practices and procedures used by the program to ready  the product design for initial manufacturing and testing as well as for  production. We also discussed the use and potential application of best  practices that we identified. In addition to discussions, we analyzed  significant amounts of data on engineering drawings, design changes, labor  efficiencies, manufacturing processes, quality indicators, testing, and  schedules. We did not verify the accuracy of the data but did correlate it to  other program indicators for reasonableness. Our analysis of the data was  used as a basis to develop indicators of each program’s development  efficiencies and detailed questions to discuss product design and  manufacturing practices.
We conducted our review between May 2001 and April 2002 in accordance  with generally accepted government auditing standards.
section_title: Timely Design and Manufacturing Knowledge Is Critical to Program Success
paragraphs: The success of any effort to develop a new product hinges on having the  right knowledge at the right time. Every program eventually achieves this  knowledge; however, leading commercial companies we visited have found  that there is a much better opportunity to meet predicted cost, schedule,  and quality targets when it is captured early, in preparation for critical  decisions. Specifically, knowledge that a product’s design is stable early in  the program facilitates informed decisions about whether to significantly  increase investments and reduces the risk of costly design changes that can  result from unknowns after initial manufacturing begins. This knowledge  comes in the form of completed engineering drawings before transitioning  from the system integration phase to the system demonstration phase of  product development. Best practices suggest that at least 90 percent of the  drawings for a product’s design be completed before a decision to commit  additional resources is made. Likewise, later knowledge that the design can  be manufactured affordably and with consistent high quality prior to  making a production decision ensures that cost and schedule targets will  be met. This knowledge comes in the form of evidence from data that  shows manufacturing processes are in control and system reliability is  achievable. Leading commercial companies rely on knowledge obtained  about critical manufacturing processes and product reliability to make  their production decisions.
The Department of Defense (DOD) programs we reviewed captured  varying amounts of design and manufacturing knowledge in the form of  completed engineering drawings and statistical process control data. We  found a correlation between the amount of knowledge each captured and  their cost and schedule outcomes. Programs that were able to complete  more engineering drawings and control their critical manufacturing  processes had more success in meeting cost and schedule targets  established when they began.
subsections: section_title: DOD Programs Had Better Outcomes When Design and Manufacturing Knowledge Was Captured at Key Program Junctures
paragraphs: Conceptually, the product development process has two phases: a system  integration phase to stabilize the product’s design and a system  demonstration phase to demonstrate the product can be manufactured  affordably and work reliably. The system integration phase is used to  stabilize the overall system design by integrating components and  subsystems into a product and by showing that the design can meet  product requirements. When this knowledge is captured, knowledge point  2 has been achieved. It should be demonstrated by the completion of at  least 90 percent of engineering drawings, which both DOD and leading  commercial companies consider to be the point when a product’s design is  essentially complete. In the DOD process, this knowledge point should  happen by the critical design review, before system demonstration and the  initial manufacturing of production representative products begins. The  system demonstration phase is then used to demonstrate that the product  will work as required and can be manufactured within targets. When this  knowledge is captured, knowledge point 3 has been achieved. Critical  manufacturing processes are in control and consistently producing items  within quality standards and tolerances for the overall product. Also,  product reliability has been demonstrated. In the DOD process, like with  the commercial process, this knowledge point should happen by the  production commitment milestone. Bypassing critical knowledge at either  knowledge point will usually result in cost, schedule, and performance  problems later in product development and production.
We found that the most successful programs had taken steps to gather  knowledge that confirmed the product’s design was stable before the  design was released to manufacturing organizations to build products for  demonstration. They had most of the detailed design complete, supported  by the completion of a large percentage of engineering drawings to  manufacturing. Again, engineering drawings are critical because they  include details on the parts and work instructions needed to make the  product and reflect the results of testing. These drawings allowed  manufacturing personnel to effectively plan the fabrication process and  efficiently build production representative prototypes in the factory so  manufacturing processes and the product’s performance could be validated  before committing to production. The most successful DOD programs also  captured the knowledge that manufacturing processes needed to build the  product would consistently produce a reliable product by the end of system  demonstration, before making a production decision. On these programs,  the initial phase of production—sometimes known as low-rate initial  production—was able to focus on building operational test articles and  improving the production processes, instead of continuing the product’s  design and development.
Problematic programs moved forward into system demonstration without  the same knowledge from engineering drawings that successful cases had  captured. They increased investments in tooling, people, and materials  before the design was stable. In these programs, only a small percentage of  the drawings needed to make the products had been completed at the time  the designs were released to manufacturing organizations for building  production representative prototypes. In doing so, these programs  undertook the difficult challenge of stabilizing the designs at the same time  they were trying to build and test the products. This design immaturity  caused costly design changes and parts shortages that, in turn, caused  labor inefficiencies, schedule delays, and quality problems. Consequently,  these programs required significant increases in resources––time and  money—over what was estimated at the point each program began the  system demonstration phase.
The most problematic programs also started production before design and  manufacturing development work was concluded. In these cases, programs  were producing items for the customers while making major product  design and tooling changes, still establishing manufacturing processes, and  conducting development testing. These programs encountered significant  cost increases, schedule delays, and performance problems during  production.
Table 2 shows the relationship between design stability and manufacturing  knowledge at key junctures and the outcomes for the DOD programs we  reviewed. To measure design stability at the start of the system  demonstration phase, knowledge point 2, we determined the percentage of  the product’s engineering drawings that had been completed by the critical  design review. In DOD programs, after the critical design review, the  system design is released to manufacturing to begin building the  production representative prototypes for the system demonstration phase.  To measure producibility at the production decision, knowledge point 3, we  determined whether the critical manufacturing processes were in  statistical control at that time. We compared this information with best  practices. The cost and schedule experiences of the program since the start  of system demonstration are also shown.
As shown in the table, the AIM-9X and FA-18 E/F programs had captured a  significant amount of design knowledge at the start of system  demonstration and manufacturing knowledge by the start of production. In  each of those programs, product developers had the advantage of prior  versions of the systems. These programs came very close to meeting their  original cost and schedule estimates for product development. The other  three programs, F-22, PAC-3, and ATIRCM/CMWS, had less knowledge at  each key junctures. Their development cost and schedule results  significantly exceeded estimates. Specific details on the AIM-9X, F-22, and  ATIRCM/CMWS program experiences follow.
subsections: section_title: AIM-9X Program Experience
paragraphs: The AIM-9X program began in 1994, continuing the long-term evolution of  the AIM-9 series of short-range air-to-air missiles. In 1999, after developing  and testing a number of engineering prototype missiles, the program held a  critical design review to determine if the program was ready to begin initial  manufacturing of a production representative prototype for system  demonstration. At this review, about 95 percent of the eventual engineering  drawings were completed—a stable design by best practices. Because  AIM-9X was the next generation in this family of missiles, the program had  significant knowledge on how to produce the missile. At the 1999 critical  design review, the estimated development and production costs totaled  $2.82 billion. As of December 2001, the estimate was $2.96 billion, less than  a 5 percent increase.
section_title: F-22 Program Experience
paragraphs: The F-22 program began detailed design efforts in 1991 when it entered a  planned 8-year product development phase. In 1995, about the expected  midpoint of the phase, the program held its critical design review to  determine if the design was stable and complete. Despite having only about  a quarter of the eventual design drawings completed for the system, the  program declared the design to be stable and ready to begin initial  manufacturing. At that time, the program office had estimated the cost to  complete the development program at $19.5 billion. However, the program  did not complete 90 percent of its drawings for the aircraft until 1998,  3 years into the system demonstration phase. During the building of the  initial aircraft, several design and manufacturing problems surfaced that  affected the deliveries of major sections of the aircraft. Large sections were  delivered incomplete to final assembly and had to be built out of the  planned assembly sequence.
In 1997, an independent review team examined the program and  determined the product development effort was underestimated. The team  found that building the first three aircraft was taking substantially more  labor hours than planned. Between 1995 and 1998, the development  estimate for the F-22 increased by over $3.3 billion and the schedule  slipped by a year. Achieving design stability late has contributed to further  cost increases. As of December 2001, the estimated development cost was  $26.1 billion, a 34 percent increase since the critical design review was held  in 1995.
While the program attributes some production cost increases to a  reduction in F-22 quantities, it has been significantly affected by design and  manufacturing problems that started during development. The independent  review team evaluated the cost impact on the production aircraft that  would likely occur because of cost and schedule problems in development  and found that production aircraft would have to begin later, at a slower  pace, and cost more than expected. The team estimated that production  costs could increase by as much as $13 billion if savings were not found.  The Air Force subsequently increased the estimate to more than $19 billion  in cost savings required to avoid cost increases. In 2001, when the F-22  limited production decision was made, the program had less knowledge  about the aircraft’s reliability and manufacturing processes than more  successful cases. For example, at its limited production decision, it had  only 44 percent of its critical manufacturing processes in control. In  September 2001, the program reported that overall production cost would  likely increase by more than $5.4 billion. This estimate was based on the  effort needed so far to build the aircraft during product development.
section_title: ATIRCM/CMWS Program Experience
paragraphs: Since it began in 1995, the ATIRCM/CMWS program has had significant cost  growth and schedule delays during product development. The product  developer held a major design review in 1997. Like the F-22, the review  demanded less proof about the product’s design in the form of engineering  drawings before deciding to begin initial manufacturing. At that time, only  21 percent of the engineering drawings had been completed, and it was still  unknown whether the design would meet the requirements. In fact, the  program knew that a major redesign of a critical component was needed.  Despite this, the program office deemed the risk acceptable for moving the  program forward to begin manufacturing prototypes. Over the next 2 years,  the program encountered numerous design and manufacturing problems. It  was not until 1999, about 2 years after the critical design review, that  program officials felt that the design had stabilized; however, by this time,  the product development cost had increased 160 percent and production  had been delayed by almost 3 years.
ATIRCM/CMWS is scheduled to begin limited production in early 2002, but  without the same degree of assurance as the more successful programs  that the product can be manufactured within cost, schedule, and quality  targets. The program has not yet determined if manufacturing processes  needed to build the product are in control. Many of the development units  were built by hand, in different facilities, and with different processes and  personnel. Program officials stated that because they did not stabilize the  design until late in development, manufacturing issues were not adequately  addressed. Since 1997, the estimated unit cost for the system has increased  by 182 percent.
section_title: Best Practices Enable Timely Capture of Design and Manufacturing Knowledge
paragraphs: Leading commercial companies have been successful in achieving product  development goals because they have found ways to enable the capture of  design and manufacturing knowledge about the products they are  developing in a timely way. We found two practices that allowed leading  commercial companies to capture necessary knowledge for product  development. First, they established a framework of evolutionary product  development that limited the amount of design and manufacturing  knowledge that had to be captured. This framework limited the design  challenge for any one new product development by requiring risky  technology, design, or manufacturing requirements to be deferred until a  future generation of the product. Second, each company (1) employed a  disciplined product development process that brought together and  integrated all of the technologies, components, and subsystems required  for the product to ensure the design was stable before entering product  demonstration and (2) demonstrated the product was reliable and  producible using proven manufacturing processes before entering  production.
The product development process includes tools that both capture  knowledge and tie this knowledge to decisions about the product’s design  and manufacturing processes before making commitments that would  significantly affect company resources. For example, during system  integration, each leading commercial company used various forms of  prototypes and information from predecessor products to stabilize the  product’s design and identify critical processes, then used a decision  review that required agreements from key stakeholders that the requisite  design knowledge was captured in making a decision to move into system  demonstration. During system demonstration, each company used  statistical process control and reliability testing to ensure the product  could be produced affordably and would be reliable, then used a similar  decision review that required agreements from key stakeholders that the  requisite knowledge was captured when deciding to move into production.
The Department of Defense (DOD) programs that we reviewed used some  of these practices to varying degrees and experienced predictable  outcomes. For example, the AIM-9X and F/A-18 E/F programs were  evolutionary in nature, modifications of existing products with a  manageable amount of new technological or design challenges. They also  gathered design and manufacturing knowledge, although not to the extent  we found at commercial companies. Finally, they held program reviews and  ensured that the design and manufacturing knowledge was captured before  moving forward. They had relatively successful outcomes. The other DOD  programs—the F-22, ATIRCMS, and PAC-3—did not closely approximate  best practices in capturing design or manufacturing knowledge during  product development. They took on greater design challenges, had  program reviews that were not supported by critical design and  manufacturing knowledge, and made decisions to advance to the next  phases of development without sufficient design and manufacturing  knowledge.
subsections: section_title: Leading Commercial Companies Use Evolutionary Product Development Framework to Reduce Development Risks
paragraphs: A key to the success of commercial companies was using an evolutionary  approach to develop a product. This approach permitted companies to  focus more on design and development with a limited array of new content  and technologies in a program. It also ensured that each company had the  requisite knowledge for a product’s design before investing in the  development of manufacturing processes and facilities. Companies have  found that trying to capture the knowledge required to stabilize the design  of a product that requires significant amounts of new content is an  unmanageable task, especially if the goal is to reduce cycle times and get  the product into the marketplace as quickly as possible. Design elements  not achievable in the initial development were planned for subsequent  development efforts in future generations of the product, but only when  technologies were proven to be mature and other resources were available.
Commercial companies have implemented the evolutionary approach by  establishing time-phased plans to develop new products in increments  based on technologies and resources achievable now and later. This  approach reduces the amount of risk in the development of each  increment, facilitating greater success in meeting cost, schedule, and  performance requirements. In effect, these companies evolve products,  continuously improving their performance as new technologies and  methods allow. These evolutionary improvements to products eventually  result in the full desired capability, but in multiple steps, delivering a series  of enhanced interim capabilities to the customer more quickly.
Historically, DOD’s approach has been to develop new weapon systems  that often attempt to satisfy the full requirement in a single step, regardless  of the design challenge or the maturity of technologies necessary to  achieve the full capability. Under this single-step approach, a war fighter  can wait over 15 years to receive any improved capability. Figure 5 shows a  notional comparison between the single-step and evolutionary approaches.
Each commercial company we visited used the evolutionary approach as  the primary method of product development. General Electric builds on the  basic capability of a fielded product by introducing proven improvements  in capability from its advanced engineering development team. General  Electric considers the introduction of immature technologies into fielded  products or new engine development programs as a significant cost and  schedule risk. Its new product development process is primarily focused  on reducing and managing risk for design changes and product  introductions. Cummins and Hewlett Packard managers indicated that, in  the past, their companies learned the hard way by trying to make quantum  leaps in product performance and by including immature technologies.  Now, both companies have new product development processes that  actively manage the amount of new content that can be placed on a new  product development effort. Caterpillar also limits new content on its new  products as a way to more successfully and cost-effectively develop new,  but evolutionary, products. Even during the development of its 797 mining  truck, which it considered a major design challenge, it did not require the  truck to achieve capabilities—such as prognostics for better  maintenance—that it could not demonstrate or validate in the design in a  timely manner.
Of the five DOD programs we reviewed, two—the F/A-18-E/F and the  AIM-9X—were variations of existing products—the F/A-18-C/D and the  AIM-9M—and the programs made a commitment to use existing  technologies and processes as much as possible. These two programs had  relatively successful cost and schedule outcomes. They represented an  exception to the usual practice in DOD. The overwhelming majority of  DOD’s major acquisitions today require major leaps in capability over their  predecessors or any other competing weapon systems, with little  knowledge about the resources that will be required to design and  manufacture the systems. Decisions are continually made throughout  product development without knowing the cost and schedule  ramifications.
section_title: Leading Commercial Companies Use a Product Development Process to Capture Design and Manufacturing Knowledge for Decision Making
paragraphs: Leading commercial companies we visited had spent significant amounts of  time and resources to develop and evolve new product development  processes that ensured design and manufacturing knowledge was captured  at the two critical decision points in product development: when the  product’s design was demonstrated to be stable—knowledge point 2—and  when the product was demonstrated to be producible at an affordable  cost—knowledge point 3. The process established a disciplined framework  to capture specific design and manufacturing knowledge about new  products. Companies then used that knowledge to make informed  decisions about moving forward in a new product development program.  Commercial companies tied this knowledge to decisions about the  products’ design and manufacturing processes before making  commitments that would significantly impact company resources. Each  commercial firm we visited had a new product development process that  was prominent and central to the firm’s successes. It included three  aspects: (1) activities that led to the capture of specific design knowledge,  (2) activities that led to the capture of specific manufacturing and product  reliability knowledge, and (3) decision reviews to determine if the  appropriate knowledge was captured to move to the next phase.
subsections: section_title: Design Knowledge Should Be Captured before Entering Product Demonstration
paragraphs: To ensure that the product’s design was stable before deciding to commit  additional resources to product demonstration, commercial companies  demanded knowledge, either from existing product information or by  building engineering prototypes. They also used a disciplined design review  process to examine and verify the knowledge that had culminated at the  end of product integration, This design review process required agreement  from stakeholders that the product design could be produced and would  satisfy the customer’s requirements. Stakeholders included design  engineers, manufacturing or production personnel, and key supplier  representatives who used engineering drawings, supported by test results  and engineering data, as a key indicator of the design’s stability. Once the  program achieved a stable design, the certainty of their cost and schedule  estimates was substantially increased, allowing them to plan the balance of  the product development program with high confidence. Table 3 shows the  activities required to capture design knowledge that leads to executive  decisions about whether to transition to the next phase of development.
A key tool used by each company to ensure that a product’s design was  stable by the end of the product integration phase was a demonstration that  the design would meet requirements. The companies visited indicated that  prototypes at various system levels were the best way to demonstrate that  the product’s design would work. If the product under development was an  incremental improvement to existing products, such as the next generation  of a printer or engine, these companies used virtual prototypes for any  components that were being used for the first time. If the product included  more new content or invention, fully integrated prototypes were frequently  used to demonstrate that the design met requirements. Prototypes at this  stage in development were typically not built in a manufacturing facility.  This allowed demonstrations of the design before the companies made  more costly investments in manufacturing equipment and tooling to build  production representative prototypes for the demonstration phase. Table 4  shows an example of the types and purposes for various kinds of  prototypes used by Cummins Inc. depending on the amount of knowledge  it needed to capture and the point it was in the development process.  Prototypes were used by commercial companies throughout the product  development process and not just during product integration.
Cummins, the world sales leader in diesel engines over 200 horsepower,  effectively uses prototypes to ensure that a design is stable and believes in  the value of prototyping throughout product development. A Cummins  representative stated that not using prototypes becomes a matter of “pay  me now or pay me later,” meaning that it is far less costly to demonstrate a  product’s design early in development with prototypes, concepts, and  analyses than to incur the cost of significant design changes after a product  has entered production—a much more costly environment to make  changes. Cummins built and tested 12 engineering concept prototype  engines for its Signature 600 engine, a new concept, 600 horsepower,  overhead cam diesel engine that represented a quantum leap in  performance beyond Cummins’ existing products. These prototypes were  built using production-like tooling and methods using production workers.  In addition to using engineering prototypes during the product integration  phase of product development, Cummins and other companies we visited  used other prototypes—such as production representative prototypes—in  the remaining product development phases before production, as shown in  table 4, to demonstrate product reliability and process control. Prior to  reaching production for its Signature 600 engine, Cummins used many  prototypes to complete hundreds of thousands of test hours, accumulating  millions of test miles.
Caterpillar, a major manufacturer of heavy equipment, has a continuous  product improvement philosophy. That is, it tries to develop new products  that increase the capabilities of existing product lines, but it limits the  amount of new content on any one product development because new  content inherently increases design risk. In evolving its products this way,  Caterpillar is able to use modeling and simulation prior to initial  manufacturing because it has existing products to provide a baseline of  knowledge and a good benchmark for assessing the simulated  performance. In addition, with knowledge of existing components, it can  focus attention on maturing the new content, the higher risk element of the  new product. When Caterpillar developed the 797 mining truck, a new 360- ton payload truck design, it demonstrated design stability by identifying the  critical components and building engineering prototypes of them for  reliability testing and demonstration of the design before beginning initial  manufacturing. This knowledge, coupled with vast experience in  manufacturing trucks, ensured the stability of the 797-truck design before  initial manufacturing started. Caterpillar was able to deliver this design in  18 months after the product development was started.
subsections: section_title: Disciplined Reviews and Stakeholder Agreements Supported the Capture of Design Knowledge
paragraphs: The commercial companies we visited understood the importance of  having disciplined design reviews and getting agreement from the  stakeholders that the product’s design had been demonstrated to meet  requirements before beginning initial manufacturing. Each company had a  design review process that began at the component level, continued  through the subsystem level, and culminated with a critical design review  of the integrated system to determine if the product was ready to progress  to the next phase of development. In addition to design engineers, a cross- functional team of stakeholders in the process included key suppliers,  manufacturing representatives, and service and maintenance  representatives. From past experience, commercial companies have  discovered that cross-functional teams provide a complete perspective of  the product. While design engineers bring important skills and experience  to creating a product design, they may not be aware of manufacturing  issues, available technologies, or manufacturing processes, and they may  design a product that the company cannot afford to produce or maintain.
The product’s design is stable when all stakeholders agree that engineering  drawings are complete and that the design will work and can be built. A  commercial company considers engineering drawings to be a good  measure of the demonstrated stability of the product’s design because they  represent the language used by engineers to communicate to the  manufacturers the details of a new product design—what it looks like, how  its components interface, how it functions, how to build it, and what  critical materials and processes are required to fabricate and test it. The  engineering drawing package released to manufacturing includes items  such as the schematic of the product’s components, interface control  documents, a listing of materials, notations of critical manufacturing  processes, and testing requirements. It is this package that allows a  manufacturer to build the product in the manufacturing facility.
In developing the Signature 600, Cummins used cross-functional design  teams that included stakeholders from suppliers, machine tool  manufacturers, foundry and pattern makers, purchasing, finance,  manufacturing engineering, design engineering, and other technical  disciplines. Signature 600 components were designed with the key  suppliers co-located at the Cummins design facility. Likewise, Caterpillar  said that early supplier and manufacturing involvement was critical to  success and that engineering drawings were signed by design and  manufacturing stakeholders. Caterpillar representatives said that signing  the drawings was a certification that the design could be manufactured the  next day, if necessary.
section_title: Executive Level Reviews Were Required to Begin Initial Manufacturing
paragraphs: Each commercial company, after capturing specific design knowledge, had  an executive level review at the decision point to determine if the product  design had sufficiently progressed to permit a transition from product  integration to product demonstration. This decision point used the  knowledge captured as exit criteria for moving to the next phase of  development. For example, to demonstrate the product design was stable  and ready to move from integration to demonstration, the design had to be  demonstrated, at least 90 percent of the engineering drawings had to be  completed, design reviews had to be completed, and stakeholders had to  agree the design was complete and producible. If the design team could not  satisfy the exit criteria, then other options had to be considered. Options  included canceling the development program, delaying the decision until  all criteria were met, or moving ahead with a detailed plan to achieve  criteria not met by a specific time when leadership would revisit the other  options. One company emphasized that if a major milestone is delayed, an  appropriate adjustment should be made to the end date of the program,  thereby avoiding compressing the time allotted for the rest of product  development and managing the risks that subsequent milestones will be  missed.
This decision point coincides with the companies’ need to increase  investments in the product development and continue to the next phase.  For this reason, the decision point was considered critical to achieving  success in product development and could not be taken lightly. For  example, transitioning from the integration to the demonstration phase  requires a significant investment to start building and testing production  representative prototypes in a manufacturing environment. This requires  establishing a supplier base and purchasing materials. In addition,  establishing tooling and manufacturing capability is also required. After a  product passes this decision point and added investments are made, the  cost of making changes to the product design also increases significantly.  Therefore, commercial companies strive to firm the design as early in the  process as possible when it is significantly cheaper to make changes.
section_title: Manufacturing and Product Reliability Knowledge Should Be Captured before Starting Production
paragraphs: We found that leading commercial companies used two tools to capture  knowledge that a product’s design was reliable and producible within cost,  schedule, and quality targets before making a production decision. These  tools are (1) a quality concept that uses statistical process control to bring  critical manufacturing processes under control so they are repeatable,  sustainable, and consistently producing parts within the quality tolerances  and standards of the product and (2) product tests in operational  conditions that ensure the system would meet reliability goals–the ability  to work without failure or need of maintenance for predictable intervals.  Company officials told us that these two tools enabled a smooth transition  from product development to production, resulting in better program  outcomes. Companies employed these tools on production representative  prototypes, making the prototypes a key ingredient to successful  outcomes. Table 5 shows the activities required to capture manufacturing  knowledge that leads to executive decisions about whether to transition  from product development into production.
Commercial companies rely on statistical process control data to track,  control, and improve critical manufacturing processes before production  begins. Bringing processes under statistical control reduces variations in  parts manufactured, thus reducing the potential for defects. Product  variation has been called the “silent killer” on the manufacturing floor  because it can result in defects that require additional resources to either  rework or scrap the product. Products fielded with defects may have  degraded performance, lower reliability, or increased support costs.  Experience has taught commercial companies that it is less costly—in  terms of time and money—to eliminate product variation by controlling  manufacturing processes than to perform extensive inspection after a  product is built. Because thousands of manufacturing processes can be  required to build a product, companies focus on the critical processes— those that build parts that influence the product’s performance, service life,  or manufacturability. Therefore, when design engineers are designing the  new product, they must identify its key characteristics so that  manufacturing engineers can identify and control critical manufacturing  processes. Key product characteristics and critical manufacturing  processes are noted on the engineering drawings and work instructions  that are released to manufacturing.
Once critical processes are identified, companies perform capability  studies to ensure that a process will produce parts that meet specifications.  These studies yield a process capability index (Cpk), a measure of the  process’s ability to build a part within specified limits. The index can be  translated into an expected product defect rate. The industry standard is to  have a Cpk of 1.33 or higher, which equates to a probability that 99.99  percent of the parts built on that process will be within the specified limits.  Four of the five companies we visited wanted their critical processes at a  minimum of a 1.33 Cpk and many had goals of achieving higher Cpks. Table  6 shows various Cpk values and the defect rate associated with each value.  The table also shows the higher the Cpk, the lower the defect rate.
Cpk values also have an additive effect on various individual parts when  each part is integrated into the final product. For example, a product  composed of 25 parts, where each part is produced on a manufacturing  process with a Cpk of 0.67, has a 95.5 percent probability that each part will  be defect free. However, when all 25 parts are assembled into the final  product, the probability that the final product will be defect free is only  32 percent. In comparison, if the same parts are produced with  manufacturing processes at a Cpk of 1.33, the probability of each part being  defect free is 99.99 percent. When these same 25 parts are assembled into  the final product, the probability that the final product will be defect free is 99.8 percent. This comparison illustrates the impact that having  manufacturing processes in control has on the amount of rework and  repair that would be needed to correct defects and make the product meet  its specifications.
Cummins uses statistical process control data to measure a product’s  readiness for production. In developing the new Signature 600 diesel  engine, Cummins included manufacturing engineers and machine tool and  fixture suppliers in the design decision process as the engine concept was  first being defined. Cummins built production representative prototypes of  its engines to demonstrate that the design and the engine hardware would  perform to requirements. These prototypes represented the first attempt to  build the product solely using manufacturing personnel, production  tooling, and production processes. Cummins used the knowledge captured  from these and subsequent prototypes to refine and eventually validate the  manufacturing processes for the engine. This process of employing  statistical process control techniques on prototype engines verified that the  manufacturing processes were capable of manufacturing the product to  high quality standards within established cost and schedule targets.
Other companies we visited emphasized the importance of controlling  manufacturing processes before committing to production. For example,  Xerox captures knowledge about the producibility of its product early in  the design phase. By production, it strives to have all critical manufacturing  processes for the product—including key suppliers’ processes—in control  with a Cpk index of at least 1.33. Xerox achieves this by building  production representative prototypes and by requiring suppliers of key  components and subassemblies to produce an adequate sample of parts to  demonstrate the suppliers’ processes can be controlled, usually before the  parts are incorporated into the prototypes. General Electric Aircraft  Engines has digitally captured, and made available to design engineers, Cpk  data on almost all of its manufacturing processes and it strives to have  critical processes in control to a point where they will yield no more than  1 defect in 500 million parts, a Cpk of 2.0. Other companies, such as  Caterpillar and Hewlett Packard, told us that getting manufacturing  processes in control prior to production is key to meeting cost, schedule,  and quality targets. Each of the companies visited used this as an indicator  of the product’s readiness for production and emphasized the importance  of having critical manufacturing processes under control by the start of  production.
subsections: section_title: Demonstrating Product Reliability Indicates the Product Is Ready for Production
paragraphs: A product is reliable when it can perform over a specified period of time  without failure, degradation, or need of repair. Reliability is a function of  the specific elements of a product’s design. Making design changes to  achieve reliability requirements after production begins is inefficient and  costly. Reliability growth testing provides visibility over how reliability is  improving and uncovers design problems so fixes can be incorporated  before production begins.
In general, reliability growth is the result of an iterative design, build, test,  analyze, and fix process. Prototype hardware is key to testing for reliability  growth. Initial prototypes for a complex product with major technological  advances have inherent deficiencies. As the prototypes are tested, failures  occur and, in fact, are desired so that the product’s design can be made  more reliable. Reliability improves over time with design changes or  manufacturing process improvements. The earlier this takes place, the less  impact it will have on the development and production program.  Companies we visited matured a product’s reliability through these tests  and demanded proof that the product would meet the customer’s reliability  expectations prior to making a production decision.
Improvements in the reliability of a product’s design can be measured by  tracking a key reliability metric over time. This metric compares the  product’s actual reliability to a growth plan and ultimately to the overall  reliability goal. Several commercial companies we visited began gathering  this data very early in development and tracked it throughout development.  The goal was to demonstrate the product would meet reliability  requirements before starting full rate production.
Caterpillar establishes a plan to grow and demonstrate the product’s  reliability before fabrication of a production representative prototype  begins. Before Caterpillar starts making parts, it estimates the product’s  reliability in its current stage of development based on knowledge captured  from failure modes and effects analysis, component prototype testing, and  past product experience. This information marks the starting point for the  product’s reliability growth plan and is the basis for assessing whether the  plan is achievable by production. If Caterpillar believes the risks are too  high and the goal cannot be achieved on time, decision makers assess  trade-offs between new and existing components to reduce the risks to a  more manageable level. Trade-offs might be made if the product’s  performance still fails to meet requirements. If trade-offs are not possible,  decision makers may decide not to go forward with the development. Once  Caterpillar has established this plan, it tracks demonstrated reliability  against it as a management tool to measure progress. It sets an interim  reliability milestone and expects to be at least halfway toward the expected  goal by the time it begins to build production units. Caterpillar has learned  from experience that it will achieve the full reliability goal by full  production if it meets the interim goal by the time it produces pilot  production units. If the reliability is not growing as expected, then  decisions about changing or improving the design must be addressed.
Caterpillar improves the product’s reliability during development by testing  prototypes, uncovering failures, and incorporating design changes.  According to Caterpillar officials, the production decision will be delayed if  they are not on track to meeting their reliability goal. These officials told us  that Caterpillar maintains the philosophy of first getting the design right,  then producing it as quickly and efficiently as possible. They emphasized  that demonstrating reliability before production minimized the potential  for costly design changes once the product is fielded.
section_title: Executive Level Reviews Are Conducted to Begin Production
paragraphs: The commercial companies, after capturing specific manufacturing  knowledge, had executive level reviews to determine if the product  development had sufficiently progressed to permit a transition into  production. Executives used the knowledge captured as exit criteria for the  transition. For example, to demonstrate the product was ready for  production, critical processes had to be in control and testing should have  demonstrated the product reliability. If the design team could not satisfy  the exit criteria, then other options had to be considered. The production  decision led to increased investments for materials and resources such as  additional tooling to build the product at a planned rate, facilities, people,  training and support.
section_title: When DOD Programs More Closely Approximated Best Practices, Outcomes Were Better
paragraphs: Our analysis of DOD programs showed that those more closely  approximating best practices had better outcomes. The F/A-18 E/F fighter  and the AIM-9X missile were based extensively on predecessor programs  and employed similar tools to capture design and manufacturing  knowledge at critical program junctures. These programs had  demonstrated a significantly higher degree of design stability prior to  entering system demonstration and committing to initial manufacturing  when compared to other DOD weapon programs in our review. They also  gained control of most of their manufacturing processes and demonstrated  that the products were reliable before entering production. The success of  these programs is best demonstrated by the fact that they have been close  to meeting cost, schedule, and performance objectives. On the other hand,  the PAC-3 missile, F-22 fighter, and ATIRCM/CMWS programs did not use  these best practices. These programs were not based on predecessor  products or evolutionary in nature, and each product’s full capability was  expected in one step, with the first product off the production line. With  this daunting task, these programs failed to demonstrate a stable design  before committing to initial manufacturing, causing quality and labor  problems. These programs also had much less knowledge about the  manufacturability of their design when they entered production. As a  result, they experienced significant increases in development costs and  production delays usually at the expense of other DOD programs. Details  on the five DOD programs follow.
subsections: section_title: AIM-9X Missile Program
paragraphs: The AIM-9X development practices closely paralleled best practices used  by the commercial companies we visited. The program achieved design  stability before moving into system demonstration by incorporating mature  technologies and components from other missiles and munitions, using  engineering prototypes to demonstrate the design, holding a series of  design reviews prior to the system level critical design review, and  completing and releasing 95 percent of the engineering drawings at that  time. Figure 6 shows the building of knowledge required to achieve a stable  design on the AIM-9X.
The AIM-9X program made extensive use of engineering prototypes to  stabilize the missile’s design before building production representative  prototypes. Program officials stated that testing of engineering prototypes  uncovered problems with missile design and manufacturing tooling early in  the development, during system integration, allowing time to re-design and  re-test in follow-on configurations. According to program officials, this not  only helped stabilize the design before entering initial manufacturing but  grew system reliability and reduced total ownership costs. The program  also held design reviews for each of the major subsystems, allowing the  program to achieve and demonstrate a stable design in July 1999, before  beginning initial manufacturing of production representative prototypes.
While the AIM-9X used statistical process control only to a limited extent,  other factors have allowed it to have a more successful production  outcome to date. Program officials took steps to ensure that manufacturing  aspects of the product were included in the design, including empowering a  product leader with a manufacturing background, identifying the key  characteristics and critical manufacturing processes early, making design  trade-offs to enhance manufacturing capability, and demonstrating a robust  design to make the product less vulnerable to variations in manufacturing  process. In addition, the ability to achieve design stability at the critical  design review allowed program officials to focus the system demonstration  phase on maturing the manufacturing processes. Prior to committing to  production, the program demonstrated that the product could be efficiently  built using production processes, people, tools, and facilities to build  prototypes. According to the former program manager, these steps gave the  officials knowledge that a reliable product could be produced within cost  and schedule targets prior to entering production. To date, the AIM-9X  program has largely met its production targets.
section_title: F/A-18 E/F Program
paragraphs: The F/A-18 E/F aircraft development program was able to take advantage  of knowledge captured in developing and manufacturing prior versions of  the aircraft. This evolutionary approach significantly contributed to the  cost and schedule successes of this program. Because the F/A-18 E/F was a  variant of the older F/A-18 aircraft, the developer had prior knowledge of  design and manufacturing problems. This knowledge, coupled with the use  of modeling and computer-aided design software, helped create a design  that was easier to manufacture. While the program did not fully use each of  the best practices, it did embrace the concepts of capturing design and  manufacturing knowledge early in the program.
During the program’s critical design review, about 56 percent of the  drawings were completed and, while the program did not meet the best  practice of 90 percent complete, it did have additional drawing data of the  F/A-18 E/F assemblies available for review at the critical design review. The  Navy used early versions of the F/A-18 aircraft to demonstrate new  component designs and new materials. In addition, the aircraft was  designed to have 42 percent fewer parts than its predecessor, making its  design more robust. The program also identified the critical manufacturing  processes and collected statistical process control data early in product  development. At the start of production, 78 percent of these critical  processes were in control. Unit costs for the F/A-18 E/F program have not  grown since the critical design review and its schedule has been delayed by  only 3 months.
section_title: F-22 Fighter Program
paragraphs: The F-22 program is structured to provide the product’s full capability with  the first product off the production line—an extreme design challenge. This  required the product design to include many new and unproven  technologies, designs, and manufacturing processes. It did not demonstrate  design stability until about 3 years after it held its critical design review.  The program completed 3,070 initial engineering drawings at its critical  design review in 1995, about 26 percent of the eventual drawings needed. It  did not complete 90 percent of the necessary engineering drawings until  1998, after the first two development aircraft were delivered. Figure 7  shows the drawing completion history for the program.
After its critical design review, the F-22 program encountered several  design and manufacturing problems that resulted in design changes, labor  inefficiencies, cost increases, and schedule delays. For example, delivery of  the aft fuselage—the rear aircraft body section—was late for several of the  test aircraft and two ground test articles because of late parts and  difficulties with the welding process. According to the F-22 program office,  design maturity and manufacturing problems caused a “rolling wave” effect  throughout system integration and final assembly. Late engineering  drawing releases to the factory floor resulted in parts shortages and work  performed out of sequence. These events contributed to significant cost  overruns and delays to aircraft deliveries to the flight test program.
The F-22 program initially had taken steps to use statistical process control  data during development and gain control of critical manufacturing  processes by the full rate production decision. In 1998, we reported that  the program had identified 926 critical manufacturing processes and had  almost 40 percent in control 2 years before production was scheduled to  begin. Although this did not match the standard set by commercial  companies, it offered major improvements over what other DOD programs  had attempted or achieved. Unfortunately, citing budgetary constraints and  specific hardware quality problems that demanded attention, the program  abandoned this best practices approach in 2000 with less than 50 percent of  it critical manufacturing processes in control. Currently, the program is  using post-assembly inspection to identify and fix defects rather than  statistical process control techniques to prevent them. In March 2002, we  recommended that the F-22 program office monitor the status of critical  manufacturing processes as the program proceeds toward high rate  production. The program stated that it would assess the processes status as  the program moves forward.
The program entered limited production despite being substantially behind  its plan to achieve reliability goals. A key reliability requirement for the  F-22 is mean time between maintenance, defined as the number of  operating hours for the aircraft divided by the number of maintenance  actions. The reliability goal for the F-22 is a 3-hour mean time between  maintenance. The Air Force estimated that in late 2001, when the F-22  entered limited production, it should have been able to demonstrate almost  2 flying hours between maintenance actions. However, when it actually  began limited production it could only fly an average of 0.44 hours between  maintenance actions. In other words, the F-22 is requiring significantly  more maintenance actions than planned. Additionally, the program has  been slow to fix and correct problems that have affected reliability. To date,  the program has identified about 260 different types of failures, such as  main landing gear tires wearing out more quickly than planned, fasteners  being damaged, and canopy delaminating. It has identified fixes for less  than 50 percent of these failures. Ideally, the design fixes for the failures  should be corrected prior to manufacturing production units.
section_title: PAC-3 Missile Program
paragraphs: The PAC-3 missile did not achieve design stability until after the building of  production representative prototypes for system demonstration began. At  the program’s critical design review, the PAC-3 program had completed 980  engineering drawings—21 percent of the eventual drawings needed for the  missile. Since then, almost 3,700 more drawings have been completed. The  total number of drawings expected to represent the completed design grew  from about 2,900 at the critical design review to almost 4,700 as of July  2001. This uncertainty in the expected drawings not only indicates that the  design was not stable when initial manufacturing began but also shows that  there was a significant lack of knowledge about the design. Figure 8 shows  the design knowledge at the critical design review, when the decision was  made to commit to initial manufacturing of the missile.
Prototypes of the product design were not built before the critical design  review or before initial manufacturing started to show that the design  would work. Therefore, because of the immature design, initially  manufactured development missiles were hand-made, took longer to build  than planned, and suffered from poor quality. As a result, many design and  manufacturing problems surfaced during system demonstration.  Subsystems did not fit together properly, and many failed ground and  environmental tests the first time. The contractor attributed $100 million of  additional cost to first time manufacturing problems.
Prior to entering limited production in 1999, the program had less than  40 percent of the critical manufacturing processes in control for  assembling the missile and the seeker. According to program officials,  there was little emphasis during development or initial production on using  statistical control on critical manufacturing processes. Most of the  development missiles were built in specialty shops rather than in a  manufacturing environment. The result was a lack of knowledge about  whether the critical manufacturing processes could produce the product to  established cost, schedule, and quality targets. This uncertainty is reflected  in contractor estimates that more than 50 percent of the time charged to  build the initial production missiles will be for engineering activities.  Actual production labor is expected to account for about 30 percent of the  charged time.
To further understand the problems on the PAC-3 program, we focused on  its seeker subsystem, which is key to acquiring and tracking targets and  represents a large percentage of the missile’s cost. Currently, despite being  in production, it is unclear whether the supplier of the seeker can produce  it within cost, schedule, and quality targets. During development, the  supplier had difficulty in designing and manufacturing this subsystem. It  was not uncommon for seekers to be built, tested, and reworked seven or  eight times before they were acceptable. The program entered production,  despite these producibility issues. Now, even with 2 years of production  experience, the supplier continues to have difficulty producing the seeker  with acceptable quality. Data provided by the supplier in October 2001  showed that less than 25 percent of the seekers were being manufactured  properly the first time and the rest had to be reworked, on average, four  times.
section_title: ATIRCM/CMWS Program
paragraphs: According to program officials, ATIRCM/CMWS did not have a stable  design until about 2 years after the critical design review. A contributing  factor to this was a lack of understanding about the full requirements for  the new system at the critical design review in 1997. This led to a major  redesign of the common missile warning system’s sensor. At the critical  design review, only 21 percent of a product’s engineering drawings had  been completed. It did not complete 90 percent drawings—the best  practice—until 1999. The immature design caused inefficiencies in  manufacturing, rework, and delayed deliveries. In addition, between 1995  and 1999, the development contract target price increased by 165 percent.
The ATIRCM/CMWS program did not begin reliability growth testing until 4 years after its critical design review, leaving only 1 year to test the system  prior to scheduled production. Program officials said that an immature  design limited their ability to begin reliability testing earlier in  development. About one-third of the way through the reliability growth test  program, testing was halted because too many failures occurred in  components such as the power supply, the high voltage electrical system,  and the cooling system. According to a program official, the inability to  demonstrate system reliability contributed to a production delay of about  1 year. The program plans to build, develop, and test six additional  development units during 2002 and 2003 that will incorporate design  changes to fix the system failures. ATIRCM/CMWS plans to enter limited  production in the early part of 2002 with significantly less knowledge about  the design’s producibility than commercial companies. The contractor does  not use statistical process control and has not identified critical  manufacturing processes. A production readiness review identified the  lack of statistical process control as a major weakness that needs to be  corrected.
section_title: A Better Match of Policy and Incentives Is Needed to Ensure Capture of Design and Manufacturing Knowledge
paragraphs: The Department of Defense’s (DOD) acquisition policy establishes a good  framework for developing weapon systems; however, disciplined  adherence, more specific criteria, and stronger acquisition incentives are  needed to ensure the timely capture and use of knowledge in decision  making. DOD changed its acquisition policy to emphasize evolutionary  acquisition and establish separate integration and demonstration phases in  the product development process. Its goal was to develop higher quality  systems in less time and for less cost. However, DOD’s acquisition policy  lacks detailed criteria for capturing and using design and manufacturing  knowledge to facilitate better decisions and more successful acquisition  program outcomes. As demonstrated by successful companies, using these  criteria can help ensure that the right knowledge is collected at the right  time and that it will provide the basis for key decisions to commit to  significant increases in investment as product development moves  forward.
While the right policy and criteria are necessary to ensure a disciplined,  knowledge-based product development process, the incentives that  influence the key players in the acquisition process will ultimately  determine whether they will be used effectively. In DOD, current incentives  are geared toward delaying knowledge so as not to jeopardize program  funding. These incentives undermine a knowledge-based process for  making product development decisions. Instead, program managers and  contractors push the capture of design and manufacturing knowledge to  later in the development program to avoid the identification of problems  that might stop or limit its funding. They focus more on meeting schedules  than capturing and having the knowledge necessary to make the right  decisions at those milestones. Such an approach invariably leads to added  costs because programs are forced to fix problems late in development.
By contrast, commercial companies must develop high-quality products  quickly or they may not survive in the marketplace. Because of this, they  encourage their managers to capture product design and manufacturing  knowledge to identify and resolve problems early in development, before  making significant increases in their investment. Instead of a schedule- driven process, their process is driven by events that bring them  knowledge: critical design reviews that are supported by completed  engineering drawings and production decisions that are supported by  reliability testing and statistical process control data. They do not move  forward without the design and manufacturing knowledge needed to make  informed decisions.
subsections: section_title: Acquisition Policy Lacks Specific Implementation Criteria
paragraphs: Greater emphasis on evolutionary acquisitions and structuring the product  development process into two phases—system integration and system  demonstration—were good first steps for DOD to achieve its goals of  buying higher quality systems in less time and for lower costs. However,  DOD policy still lacks criteria to be used to capture specific design and  manufacturing knowledge and does not require the use of that knowledge  as exit criteria at key decision points to transition from system integration  to system demonstration and then into production. In three of the five DOD  program examples in chapter 3, managers decided to move forward in  development, even when developers had failed to capture design and  manufacturing knowledge to support increased investments. As a result,  these programs encountered significant increases in acquisition costs as  well as delays in delivering capabilities to the war fighter.
Table 7 illustrates key criteria used by commercial companies that are  currently lacking in DOD’s policy. The table shows the design and  manufacturing knowledge needed to make more informed decisions. The  capture of some of the important manufacturing and reliability knowledge  should begin in the integration phase in order to have the full knowledge  needed to make decisions at the end of the demonstration phase for  transitioning into production.
According to DOD’s current acquisition policy, the system integration  phase of an acquisition normally begins with the decision to launch a  program. The policy states that, during this phase, a system’s configuration  should be documented and the system should be demonstrated using  prototypes in a relevant environment. While these are noteworthy activities  and resemble best practices, the policy does not provide criteria for what  constitutes the level of knowledge required for completing this stage, nor  does it require a decision—based on those criteria—as to whether a  significant, additional investment should be made. Commercial companies  demand knowledge from virtual or engineering prototypes, 90 percent of  required engineering drawings for the product supported by test results,  demonstration that the product meets customer requirements, a series of  disciplined design reviews, and stakeholder agreement that the design is  stable and ready for product demonstration before a commitment is made  to move forward and invest in product demonstration. Under DOD’s  revised policy, it is still difficult to determine if a product should enter  product demonstration with a stable design.
DOD’s current acquisition policy also states that the system demonstration  phase begins after prototypes have been built and demonstrated in a  relevant environment during system integration. According to the policy, a  system must be demonstrated before the department will commit to  production. The low-rate initial production decision occurs after this phase  of product development. Like the end of system integration, the policy fails  to provide specific criteria for what constitutes the knowledge required to  support the decision to move into production. For example, the policy  states there should be “no significant manufacturing risks” but does not  define what this means or how it is measured. Without criteria for building  knowledge during the demonstration phase, the production decision is  often based on insufficient knowledge, creating a higher probability of  inconsistent results and cost and schedule problems. On the other hand,  commercial companies demand proof that manufacturing processes are in  control and product reliability goals are attained before committing to  production. With more specific knowledge in hand at the end of  development, decision makers can make a more informed decision to move  into production with assurances that the product will achieve its cost,  schedule, and quality outcomes.
Finally, while DOD’s policy separates product development into a two-stage  process—integration and demonstration—it does not require a decision  milestone to move from one stage to the next. The policy states that an  interim progress review should be held between the two stages, but the  review has no established agenda and no required outputs of information  unless specifically requested by the decision maker. Its purpose is to  confirm that the program is progressing as planned. On the other hand,  commercial companies consider this review a critical decision point in  their product development process because it precedes a commitment to  significantly increase their investment. Therefore, they use specific,  knowledge-based standards and criteria to determine if the product is  ready to enter the next phase and they hold decision makers accountable  for their actions. These decision reviews are mandatory and are typically  held at the executive level of the commercial firm.
Figure 9 illustrates the commercial model for knowledge to be captured  and delivered during product integration and product demonstration and  the possible application of that model to DOD’s acquisition process.  Without a similar decision review to bring accountability to the DOD  process, acquisition programs can—and do—continue to advance into  system demonstration without a stable design. As shown in our case  studies, this provides for a high probability of cost growth and schedule  delays to occur.
section_title: Incentives in the DOD Acquisition Environment Do Not Favor Capture of Design and Manufacturing Knowledge Early Enough
paragraphs: The incentives for program managers and product developers to gather  knowledge and reduce risk are also critical to DOD’s ability to adopt best  practices for product development. In DOD, incentives are centered on  obtaining scarce funding on an annual basis in a competitive environment  to meet predetermined and typically optimistic program schedules. These  incentives actually work against the timely capture of knowledge, pushing  it off until late in the process to avoid problems that might keep a program  from being funded. Because design and manufacturing knowledge is not  captured, key decision points intended to measure and ensure that a  weapon system has sufficiently matured to move forward in the process  risk becoming unsupported by critical knowledge. In leading commercial  companies, the opposite is true. Because companies know they have to  deliver high-quality products quickly and affordably, they limit the  challenge for their program managers and provide strong incentives to  capture design and manufacturing knowledge early in the process.  Program managers are empowered to make informed decisions before big  investments in manufacturing capability are required.
DOD’s current acquisition environment is driven by incentives to make  decisions while significant unknowns about the system’s design and  manufacturability persist. This environment results in higher risks and a  greater reliance on cost-reimbursement contracts for longer periods of  time during product development. Because events that should drive key  decisions, such as critical design reviews, interim progress reviews, and  production decision reviews, are based on inadequate design and  manufacturing knowledge, they do not support decisions to invest more  and move to the next phase of the acquisition process. Nevertheless, this  approach has proven effective in securing funds year to year. For example,  the F-22, PAC-3, and ATIRCMS/CMWS programs had less than one-third of  their engineering drawings completed at their critical design review, but  each obtained the funding necessary to move onto the initial manufacturing  of production representative prototypes. That funding allowed a significant  increase in investment to develop a manufacturing capability before critical  knowledge had been captured. The incentive to capture funding for the  program was greater than the incentive to wait, capture knowledge, and  reduce the risk of moving forward. Each of these programs encountered  significant cost increases and schedule delays.
The incentives are quite different for leading commercial companies. For  them, the business case centers on the ability to produce a product that the  customer will buy and that will provide an acceptable return on  investment. If the firm has not made a sound business case, or has been  unable to deliver on one or more of the business case factors, it faces a very  real prospect of failure—the customer may walk away. Also, if one product  development takes more time and money to complete than expected, it  denies the firm opportunities to invest those resources in other products.  For these reasons, commercial companies have strong incentives to  capture product knowledge early in the process to assess the chances of  making the business case and the need for further investments.
Production is a dominant concern in commercial companies throughout  the product development process and forces discipline and trade-offs in the  design process. This environment encourages realistic assessments of risks  and costs since doing otherwise would threaten the business case and  invite failure. For the same reasons, the environment places a high value on  knowledge for making decisions. Program managers have good reasons to  identify risks early, be intolerant of unknowns, and not rely on testing late  in the process as the main vehicle for discovering the performance  characteristics of the product. By adhering to the business case as the key  to success, program managers in leading commercial companies are  conservative in their estimates and aggressive in risk reduction. Ultimately,  adherence to the business case strengthens the ability to say “no” to  pressures to accept high risks and unknowns. Practices such as  prototyping, early manufacturing and supplier involvement, completing 90  percent of engineering drawings by critical design review, demonstrating  product reliability, and achieving statistical control of critical  manufacturing processes by production are adopted because they help  ensure success.
In DOD’s current acquisition environment, the customer is willing to trade  time and money for the highest performing weapon system possible. That  willingness drives the business case. This creates strong incentives for the  program office to take significant risks with technologies and designs to  ensure it can offer the customer a weapon system that is a quantum leap  above the competition. In addition, because funding is secured on an  annual basis in DOD, strong incentives exist for the program office to make  optimistic assumptions about development cost and schedule. Because the  customer is willing to wait and funding is never certain, an environment  exists where program managers have good reasons to avoid the capture of  knowledge and delay testing. Since the business case in DOD places very  little premium on meeting cost and schedule targets, but a very high  premium on performance, programs succeed at the point where sunk costs  make it difficult—if not prohibitive—for decision makers to cancel them.
The practices commercial companies use to capture knowledge are not  currently used in this environment because the business case does not  favor them. Instead, DOD’s product development environment relies on  cost-type contracting throughout the entire product development process.  Once in production, programs will cut quantities to maintain funding or  once fielded, they rely on the operations and maintenance budget to pay for  reliability problems not solved in development.
section_title: Conclusions and Recommendations
subsections: section_title: Conclusions
paragraphs: The Department of Defense’s (DOD) planned $700 billion investment in  weapon systems over the next 5 years requires an approach that keeps  cost, schedule, and performance risks to a minimum. This approach means  adopting and implementing an evolutionary approach to developing new  weapon systems, improving policy to more closely approximate a  knowledge-based product development process, and creating incentives  for capturing and using knowledge for decision making. Without an  evolutionary approach as its foundation, the ability to capture design and  manufacturing knowledge early in the development process is significantly  reduced. Programs, in turn, take on too much new unproven content to  meet their objectives and risks invariably increase. DOD has made  improvements in its acquisition policy by incorporating guidance for  evolutionary acquisition, creating guidelines for the development of a basic  product that can be upgraded with additional capabilities as technologies  present themselves. However, evolutionary acquisition has yet to be  consistently implemented with success on individual weapon system  acquisitions.
Regardless of whether DOD emphasized greater use of evolutionary  acquisition, acquisition programs are not capturing sufficient design and  manufacturing knowledge to make good decisions at key investment  points. The current policy establishes a good framework to develop a  product, but the policy still lacks specific criteria required to move a  program forward and does not tie knowledge to decisions for increasing  investments in the program as it moves from system integration to system  demonstration. As a result, programs often pass through each development  phase and into production with an unstable design and insufficient  knowledge about critical manufacturing processes and product reliability.  This results in greater likelihood for inconsistent and poor results and cost  and schedule problems later in the program.
Additionally, DOD does not provide the proper incentives to encourage the  use of best practices in capturing knowledge early in its development  programs. Currently, managers are focused more on the annual exercise of  obtaining funding needed to keep their programs viable and alive. The  importance of capturing design and manufacturing knowledge early gives  way to the pressures of maintaining funding, often resulting in the  acceptance of greater risks. Raising problems on a program early because  design and manufacturing knowledge is discovered can cause extra  oversight and questions that threaten a system’s survival. The prevailing  culture is to accept greater risks upfront and then fix problems later in the  development program.
We found that leading commercial companies over the years had found  ways to overcome these problems and had identified best practices that  resulted in the early capture of and use of design and manufacturing  knowledge. This was done by a combination of four key elements. First,  they established and used an evolutionary approach to develop products  that made the capture of design and manufacturing knowledge a more  manageable task. This framework limited the design challenge for any one  new product development by allowing risky technology, design, or  manufacturing requirements to be deferred until a future generation of the  product. DOD’s current policy addresses this; however, it has not had  sufficient time to show how this will be implemented.
Second, each company we visited used the same basic product  development process and criteria for bringing together and integrating all  of the technologies, components, and subsystems required for the product  to ensure the design was stable and then demonstrating that the product  was producible and reliable using proven manufacturing processes. DOD’s  policy lacks the criteria to measure design stability and process controls.  Third, successful companies used tools to capture design and  manufacturing knowledge about the product and decide about whether to  invest further based on that knowledge. Their new product development  process included key, high-level decision points before moving into product  demonstration, and again before making the production decision that  required specific, knowledge-based exit criteria. DOD’s policy does not  require a decision to move from system integration to system  demonstration. Finally, leading companies created an environment for their  managers that emphasized capturing design and manufacturing knowledge  early, before committing substantial investments in a product development  that made cancellation a more difficult decision to make. DOD’s  environment encourages meeting schedule milestones instead of capturing  design and manufacturing knowledge to make decisions.
section_title: Recommendations for Executive Action
paragraphs: DOD should take steps to close the gaps between its current acquisition  environment and best practices. To do this, it should ensure that its  acquisition process captures specific design and manufacturing  knowledge, includes decisions at key junctures in the development  program, and provides incentives to use a knowledge-based process. Such  changes are necessary to obtain greater predictability in weapon system  programs’ cost and schedule, to improve the quality of weapon systems  once fielded, and to deliver new capability to the war fighter faster. More  specifically, we recommend that the Secretary of Defense:  Require the capture of specific knowledge to be used as exit criteria for  decision making at two key points—when transitioning from system  integration to system demonstration and from system demonstration  into production. The knowledge to be captured when moving from  system integration into system demonstration should include the  following:  Completed subsystem and system design reviews.
Ninety percent of drawings completed.
Demonstration that design meets requirements—prototype or variant  testing.
Stakeholders’ (cross functional design team that includes design  engineers, manufacturing, key supplier) assurance that drawings are  complete.
Completed failure modes and effects analysis.
Identification of key system characteristics.
Identification of critical manufacturing processes.
Set reliability targets and growth plan.
The knowledge to be captured when moving from system demonstration  into production should include the following:  Demonstrated manufacturing processes.
Built production representative prototypes.
Tested prototypes to achieve reliability goal.
Tested prototypes to demonstrate product in operational  environment.
Collected statistical process control data.
Demonstration that critical processes are capable and in control.
Require that the interim progress review, currently identified in DOD’s  policy as that point in the process between system integration and  system demonstration, be a mandatory decision review. At this point,  the design should be demonstrated to be stable so that during the next  phase of development attention can be focused on demonstrating  manufacturing processes and product reliability. The program manager  should have proof—based on the exit criteria for moving out of system  integration in the above recommendation—that the product design is  stable. The exit criteria should be demonstrated and verified by the  program manager before the program can make the substantial  investments needed to begin manufacturing production representative  prototypes in the next phase of development—system demonstration.  To ensure visibility of demonstrated exit criteria to decision makers, the  criteria and the program’s status in achieving them should be included in  each program’s Defense Acquisition Executive Summary and Selected  Acquisition Reports. If the program does not meet the exit criteria,  investments should be delayed until such time as the criteria are  satisfied. To proceed without completing the required demonstrations  should require approval by the decision authority.
Expand exit criteria for the Milestone C decision to include the  knowledge to be captured during the system demonstration phase as  identified in recommendation one. This will require that the program  office demonstrate that the critical manufacturing processes are under  statistical control and that product reliability has been demonstrated  before entering production of the new weapon system. These are best  practices and indicate that the product design is mature and the  program is ready to begin production of units for operational use that  will meet the cost, schedule, and quality goals of the program.
To ensure that contracts support a knowledge-based process, we further  recommend that DOD structure its contracts for major weapon system  acquisitions so that (a) the capture and use of knowledge described in  recommendation one for beginning system demonstration is a basis for  DOD’s decision to invest in the manufacturing capability to build initial  prototypes and (b) the capture and use of manufacturing and reliability  knowledge discussed in recommendation one for moving from system  demonstration to production is a basis for DOD’s decision to invest in  production.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD concurred with a draft of this report and agreed with the benefits of  using design and manufacturing knowledge to make informed decisions at  key points in a system acquisition program. DOD had some comments with  regard to the details contained in the recommendations, which are  summarized below. DOD concurred with our recommendation to add exit  criteria at two key points in the acquisition process—when transitioning  from system integration to system demonstration and from system  demonstration into production. DOD believes, however, that the milestone  decision authority needs to retain flexibility in applying the knowledge  requirement for drawings. Flexibility and judgment are management  prerogatives that should exist in any decision process. We agree there may  be circumstances, such as in the development of software, when it makes  good sense to progress with less than the best practice standard for  drawings, but the DOD policy should maintain the requirement to achieve  90 percent drawings by the completion of the system integration phase.
DOD also concurred that critical manufacturing processes must be  demonstrated using statistical process control techniques before  production, but believes that achieving this at Milestone C, the low rate  production decision, is unlikely. It believes the criteria would be better  applied to the full rate production decision or when low rate production  quantities extend beyond 10 percent of the planned weapon system buy.   This is a reasonable approach when processes are new or unique.   However, not all critical processes will be new or unique to a specific  weapon system. Some will have been used to manufacture parts or  components for other systems or products.  At a minimum, it should be  possible to demonstrate these by Milestone C. For other critical processes  that may require additional production experience to bring under statistical  process control, a program manager should have a reasonable plan at the  Milestone C decision review to bring those processes into control by the  full rate production decision, but no later than completion of 10 percent of  the planned buy.