Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed
Published: 2010-04-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Cost growth and schedule delays are prevalent problems in acquiring defense weapon systems. Manufacturing systems has proven difficult, particularly as programs transition to production. In December 2008, the Department of Defense (DOD) issued an updated version of its acquisition policy that reflects earlier consideration of manufacturing risks. A joint defense and industry group developed manufacturing readiness levels (MRL) to support assessments of manufacturing risks. Use of MRLs on all weapon acquisition programs has been proposed. In response to a congressional request, this report assesses the manufacturing problems faced by DOD, how MRLs can address manufacturing problems, how MRLs compare to manufacturing best practices of leading commercial firms, and challenges and barriers to implementing MRLs at DOD. In conducting our work, we contacted DOD, military services, and contractors; held interviews with leading commercial firms; reviewed program documents and policy proposals; and spoke with manufacturing experts.
section_title: What GAO Found
paragraphs: DOD faces problems in manufacturing weapon systems--systems cost far more and take much longer to build than estimated. Billions of dollars in cost growth occur as programs transition from development to production, and unit-cost increases are common after production begins. Several factors contribute to these problems including inattention to manufacturing during planning and design, poor supplier management, and a deficit in manufacturing knowledge among the acquisition workforce. Essentially, programs did not identify and resolve manufacturing risks early in development, but carried risks into production where they emerged as significant problems. MRLs have been proposed as new criteria for improving the way DOD identifies and manages manufacturing risks and readiness. Introduced to the defense community in 2005, MRLs were developed from an extensive body of manufacturing knowledge that includes defense, industry, and academic sources. An analysis of DOD's technical reviews that assesses how programs are progressing show that MRLs address many gaps in core manufacturing-related areas, particularly during the early acquisition phases. Several Army and Air Force centers that piloted MRLs report these metrics contributed to substantial cost benefits on a variety of technologies and major defense acquisition programs. To develop and manufacture products, the commercial firms we visited use a disciplined, gated process that emphasizes manufacturing criteria early in development. The practices they employ focus on gathering sufficient knowledge about the producibility of their products to lower risks, and include stringent manufacturing readiness criteria to measure whether the product is sufficiently mature to move forward in development. These criteria are similar to DOD's proposed MRLs in that commercial firms (1) assess producibility at each gate using clearly defined manufacturing criteria to gain knowledge about manufacturing early, (2) demonstrate manufacturing processes in a production-relevant environment, and (3) emphasize relationships with critical suppliers. However, a key difference is that commercial firms, prior to starting production, require their manufacturing processes to be in control--that is, critical processes are repeatable, sustainable, and consistently producing parts within the quality standards. DOD's proposed MRL criteria do not require that processes be in control until later. Acceptance of MRLs has grown among some industry and DOD components. Yet, DOD has been slow to adopt a policy that would require MRLs across DOD. Concerns raised by the military services have centered on when and how the MRL assessments would be used. While a joint DOD and industry group has sought to address concerns and disseminate information on benefits, a consensus has not been reached. If adopted, DOD will need to address gaps in workforce knowledge, given the decrease in the number of staff in the production and manufacturing career fields.

section_title: Letter
paragraphs: The Department of Defense (DOD) has a well-documented history of  taking much longer and spending much more than originally planned to  develop and acquire its weapons systems. In particular, as systems  transition from development to production, programs experience  significant manufacturing problems. While DOD has made some progress  over the last two decades in addressing the problem—including policy  changes and advocating the use of best practices for product  development—GAO’s recent weapon system reviews show that  manufacturing problems, among others, continue to hinder acquisition  cost, schedule, and performance outcomes.
It is essential to find better ways of doing business and, in particular, to  make sure systems are manufactured on time and cost-effectively. To this  end, leading commercial companies have achieved more predictable  outcomes from their manufacturing efforts because they understand  producibility—the relative ease of producing designs of an item, product,  or system economically with available production techniques—and  identify manufacturing risks early and manage them effectively throughout  a product’s development life cycle.
On December 8, 2008, DOD issued a revised version of its policy  instruction on operation of the defense acquisition system that, among  other things, recognizes the need to consider manufacturing risks earlier  in the acquisition life cycle and assesses risks prior to key decision points.  In response to the need for the department to better design and produce  more affordable weapon systems, and to give decision makers and  managers better visibility into their program risks, a joint defense and  industry working group was established in 2004 to develop manufacturing  readiness levels (MRL), a measurement scale designed to improve the  management and communication of manufacturing risk and readiness  throughout acquisitions. Similar to technology metrics that measure the  readiness of a technology, MRLs are new manufacturing criteria that  measures the manufacturing maturity or readiness of a given technology,  manufacturing process, system, or element of a weapon system at various  phases of the acquisition life cycle.
In response to a request from the Senate Subcommittee on Emerging  Threats and Capabilities and Senator Reed, we reviewed DOD’s efforts to  adopt MRLs. This report addresses (1) the manufacturing problems  experienced by selected DOD programs, (2) how MRLs can address DOD’s  manufacturing problems, (3) how proposed MRLs compare to  manufacturing best practices of leading commercial companies, and (4)  the challenges and barriers to implementing MRLs at DOD.
To meet these objectives, we compared the manufacturing practices of  DOD and its large prime contractors with those of leading commercial  companies. We performed an aggregate analysis of DOD programs from  our annual weapons assessment. We also evaluated four major defense  weapon systems in production with known cost and schedule problems to  gain in-depth insights as to the nature and causes of problems. We also  evaluated two defense systems known to be producing systems within  cost and schedule goals and compared their practices to those employed  by commercial firms. We examined program documentation and policy  proposals, and held discussions with manufacturing and systems- engineering officials from DOD program offices, prime contractors, and  the Defense Contract Management Agency. We also reviewed lessons  learned from DOD programs that pilot-tested MRLs. We met with officials  from the Office of the Secretary of Defense, Air Force, Army, and Navy,  Missile Defense Agency, Joint Defense Manufacturing Technology Panel,  Defense Acquisition University, National Center for Advanced  Technologies, and National Defense Industrial Association to discuss  manufacturing topics and MRLs. On manufacturing workforce issues, we  interviewed officials responsible for planning activities within each of the  military services and the Defense Management Contract Agency. We  compared manufacturing and production considerations in the prior   version of DOD’s policy instruction on operation of the defense acquisition  system to those in the current version of the policy instruction.
To identify manufacturing best practices of leading commercial  companies, we interviewed and obtained documentation from  manufacturing, quality, and supplier personnel at five companies, and  reported on four companies: GE Aviation, an aerospace company; GE  Healthcare, a producer of healthcare products and services; Honeywell  Aerospace, a provider of aircraft integrated avionics, engines, systems, and  services; Siemens Mobility, a producer of light rail cars. We selected  companies that manufacture complex products and have won awards for  quality manufacturing. Appendix I includes additional details about our  scope and methodology. We conducted this performance audit from  January 2009 to February 2010 in accordance with generally accepted  government auditing standards. These standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives.
We are making recommendations to the Secretary of Defense to require an  assessment of the manufacturing readiness across DOD programs using  MRL criteria, examine strengthening the MRL criteria related to process  capability and control, assess analytical model needs and tools to support  MRL assessments, and assess the manufacturing workforce knowledge  and skills base and develop a plan to address DOD’s current and future  workforce knowledge gaps. In commenting on a draft of this report, DOD  partially concurred with the first recommendation, and concurred with the  other recommendations.
subsections: section_title: Background
paragraphs: In recognition of the lack of manufacturing knowledge at key decision  points and the need to develop more affordable weapon systems, DOD  made recent changes to its policy. In 2008, the department made  constructive changes to its policy instruction on operation of the defense  acquisition system. It also developed MRLs as a measure that could  strengthen the way the department manages and develops manufacturing- intensive systems. In 2004, the Joint Defense Manufacturing Technology  Panel sponsored a joint defense and industry working group to design and  develop MRLs for programs across DOD. In May 2005, MRLs were first  introduced to the defense community in DOD’s Technology Readiness  Assessment Deskbook for science and technology and acquisition  managers to consider.
As new manufacturing readiness criteria, MRLs are a measurement scale  designed to provide a common metric and vocabulary for assessing  manufacturing maturity and risk. MRL assessments identify the risks and  manufacturing readiness of a particular technology, manufacturing  process, weapon system, subsystem, or element of a legacy program at key  milestones throughout the acquisition life cycle. There are 10 basic MRLs  designed to be roughly congruent with comparable levels of technology  readiness levels for ease of use and understanding. Table 1 shows the  MRLs and basic definitions (see appendix II for the detailed MRL  definitions).
The working group also developed a set of elements called “threads” to  provide acquisition managers and those conducting assessments an  understanding of the manufacturing risk areas (see table 2). For these  threads, desired progress is defined for each MRL, to provide an  understanding of risks as readiness levels increase from one MRL to the  next. Conceptually, these threads are manufacturing elements that are  essential to programs as they plan, prepare for, and manage the activities  necessary to develop a product. For example, the materials thread requires  an assessment of potential supplier capability by MRL 3 and an assessment  of critical first-tier suppliers by MRL 7. Likewise, the manufacturing  personnel thread calls for identifying new manufacturing skills by MRL 3  and identifying manufacturing workforce requirements for the pilot line by  MRL 7.
As shown, each basic thread (risk area) has a description and general  requirements for assessing risks for each thread. The working group  further decomposed these MRL threads into subthreads to provide users a  detailed understanding of the various kinds of manufacturing risks. See  appendix III for a detailed breakdown of these threads (risk areas) for  each MRL.
subsections: section_title: DOD’s Long-standing History of Manufacturing Problems
paragraphs: GAO has conducted an extensive body of work that highlights many of the  manufacturing-related problems that both DOD and its prime contractors  have faced. In many respects, DOD has recognized the nature of these  problems throughout the years and has taken a number of proactive steps  to address them. GAO’s work has drawn on lessons learned and best  practices to recommend ways for DOD to improve the way it develops and  manufactures its weapon systems. Examples from our reports include the  following:  In 1996, GAO reported the practices that world-class commercial  organizations had adopted to more efficiently produce quality  products, to improve DOD’s quality assurance program. DOD was  spending $1.5 billion extra per year on military-unique quality  assurance requirements for major acquisitions and billions more on  cost and schedule overruns to correct problems. GAO concluded that  repeated unstable designs, poor process controls, and poor transition  to production caused the manufacturing quality problems. While DOD  had taken some actions, its culture was cited as the biggest reason for  slow adoption and unimplemented recommendations.
In 1998, GAO reported on best commercial practices to offer ways to  improve the process DOD uses to manage suppliers engaged in  developing and producing major weapon systems. In assessing  defense contractors and two case studies of munitions programs, the  report concluded that suppliers were critical in the amount of  technological innovation they contribute to the final product.
In 2002, GAO reported on how best practices could offer  improvements to the way DOD develops new weapon systems,  primarily the design and manufacturing aspects of the acquisition  process. DOD’s record showed a history of taking longer and  spending more than planned to develop and acquire weapon system which reduced its buying power. The report identified and  recommended best practices for capturing and using design and  manufacturing knowledge early and new development processes that s,  included high-level decision points and knowledge-based exit criter before key decisions on production are made. Essentially, one of the  high-level decision points has become what GAO commonly refers to  as Knowledge Point 3—the point when a program has demonstrated  the manufacturing processes are mature. The report also  recommended a best practice that includes a standard called the  Process Capability Index (Cpk), a process performance measurement  that quantifies how closely a product is running to its specification  limits. The index indicates how well the processes statistical  performance meets its control limit requirement.
In 2008, GAO reported on how DOD and its defense contractors can  improve the quality of major weapon systems. We reported that if  DOD continued to employ the same acquisition practices as it has in  the past, the cost of designing and developing its systems could  continue to exceed estimates by billions of dollars. Quality problems  were identified as the cause for cost overruns, schedule delays, and  reduced weapon-system availability. Like DOD prime contractors,  leading commercial firms rely on many practices related to systems  engineering, manufacturing, and supplier quality, but they were more  disciplined, and had institutionalized processes to ensure quality.
Since 2003, GAO has issued a series of annual assessment reports on  selected weapons programs, increasing from 77 to 96 programs  reviewed. At $296 billion, the cumulative cost growth for DOD  programs reported in 2009 was found to be higher than it had been five  years earlier, and the overall performance of weapon system programs  was still poor. Although the cost growth and the 22-month average  delay in delivering initial capabilities was not attributed to  manufacturing alone, the lack of production maturity was cited as one  of three key knowledge areas contributing to the department’s cost  growth, schedule delay, and performance problems.
section_title: Revised Policy Incorporates Manufacturing Best Practices
paragraphs: DOD’s December 2008 revision to its policy instruction on operation of the  defense acquisition system incorporates a number of the best practices  we identified in our previous work. The instruction covers the entire life  cycle and considers manufacturing risks earlier in the acquisition life-cycle  framework. In a November 2003 report on DOD’s May 2003 revision to its  policy, we reported that much of the revised policy agrees with GAO’s  extensive body of work and that of successful commercial firms. While we  assessed DOD’s revised policy as providing a good framework for  capturing knowledge about critical technologies, product design and  manufacturing processes, we reported in 2006 that acquisition officials  were not effectively implementing the acquisition policy’s knowledge- based process. We reported that the effective implementation of policy  was limited by the absence of effective controls that require compliance  and specific criteria for clearly demonstrating that acceptable levels of  knowledge about technology, design, and manufacturing have been  attained at critical junctures before making further investments in a  program. We concluded that without specific criteria—or standards  against which a judgment or decision is quantifiably based—decision  makers are permitted to make decisions on the basis of subjective  judgment. The December 2008 revised policy instruction establishes target  maturity criteria for measuring risks associated with manufacturing  processes at milestone decision points.
During the material solutions phase, prior to milestone A, the 2008 policy  instruction requires the analysis of alternatives to assess “manufacturing  feasibility.” During the technology development phase, prior to milestone  B, the instruction states the following:    Prototype systems or appropriate component-level prototyping shall  be employed to “evaluate manufacturing processes.”
A successful preliminary design review will “identify remaining design,  integration, and manufacturing risks.”
A program may exit the technology development phase when “the  technology and manufacturing processes for that program or  increment have been assessed and demonstrated in a relevant  environment” and “manufacturing risks have been identified.”
After milestone B, one of the purposes of the engineering and  manufacturing development phase is to “develop an affordable and  executable manufacturing process.” The instruction says that: “the  maturity of critical manufacturing processes” is to be described in a post- critical design review assessment; system capability and manufacturing  process demonstration shall show “that system production can be  supported by demonstrated manufacturing processes;” and the system  capability and manufacturing process demonstration effort shall end,  among other things, when “manufacturing processes have been effectively  demonstrated in a pilot line environment, prior to milestone C.”
Finally, at milestone C, the instruction establishes two entrance criteria for  the production and deployment phase, which include “no significant  manufacturing risks” and “manufacturing processes  under control (if  Milestone C is full-rate production).” Low-rate initial production follows in  order to ensure an “adequate and efficient manufacturing capability.” In  order to receive full-rate production approval, the following must be  shown:  1. “demonstrated control of the manufacturing process,”  2. “the collection of statistical process control data,” and  3. “demonstrated control and capability of other critical processes.”
Even with the updated policy instruction in place that includes guidance  for most knowledge-based practices, inconsistent implementation has  hindered DOD’s past efforts to reform its acquisition practices. For  example, we reported in 2006 that DOD was not effectively implementing  the knowledge-based approach process and evolutionary approach  emphasized in its policy. While the policy outlined specific knowledge- based process of concept refinement and technology development to help  ensure a sound business case is developed before committing to a new  development program, we found that almost 80 percent of the programs  we reviewed were permitted to bypass this process.
section_title: Manufacturing Problems Are Attributed to Several Factors during the Planning and Design Phases of Selected DOD Weapons Programs
paragraphs: Defense acquisition programs continue to have problems manufacturing  weapon systems. As a result, systems cost far more and take far longer to  produce than estimated. Many programs authorized to enter production  experienced billions of dollars in cost growth after the authorization— nearly two-thirds of those programs reported increases in average  procurement unit costs. Several factors contribute to these issues during  the planning and design phases. These include the inattention to  manufacturing during planning and design, poor supplier management,  and lack of a knowledgeable manufacturing workforce. Essentially, some  of these programs moved into production without considering  manufacturing risks earlier in development. This hindered managers from  later managing those risks until they became problematic, and also led to  subsequent problems with supplier management, such as prime  contractors conducting little oversight of suppliers. Some programs also  had an inadequate workforce—in terms of insufficient knowledge and  numbers—to effectively manage and oversee defense manufacturing  efforts.
subsections: section_title: Manufacturing Contributed to Growth in Cost and Delays in Schedule
paragraphs: Defense acquisition programs continue to be troubled by unstable  requirements, immature technology, and a lack of manufacturing  knowledge early in design, resulting in more costly products that take  longer to produce. Our 2009 annual assessment shows that total research  and development costs were 42 percent higher than originally estimated.   These higher costs reflect in part the learning that takes place as  manufacturing processes are established and used to produce the first  prototypes.
Even programs that have been authorized to begin production have  experienced substantial cost growth after the production decision.  Production performance can be measured by examining the cost growth  as expressed in changes to average procurement unit cost. This represents  the value DOD gets for the procurement dollars invested in a certain  program and shows the net effect of procurement cost growth and  quantity changes. Figure 1 shows the levels of average procurement unit- cost growth for selected major defense acquisition programs.
As indicated in figure 1, nearly two-thirds of programs that entered  production after 2000 reported more than a 5 percent increase in average  unit cost growth, while 32 percent of programs reported average unit cost  growth that ranged from 11 percent to more than 15 percent. One program  reported a 25 percent increase in average procurement unit cost. Further,  42 percent of those programs experienced production cost increases when  procured quantities decreased or remained the same. For example, the  Black Hawk helicopter’s 2007 production estimate had no increase in  quantities since 2005, yet its production cost increased $2.3 billion, and  average procurement unit cost rose by 13 percent. The Joint Air-to-Surface  Standoff Missile had an 8 percent quantity decrease since the 2004  production decision; but the production costs increased by $561 million  and average procurement unit cost increased by 25 percent.
As for schedule growth, DOD has continued to experience delays in  delivering new or modified weapon systems to the warfighter. Over 50  percent of current programs in production have encountered some form of  delay after the production decision, when manufacturing processes should  be in control. Consequently, warfighters often must operate costly legacy  systems longer than expected, find alternatives to fill capability gaps, or go  without the capability altogether.
The four DOD weapon systems we selected for in-depth review with  known cost, schedule, and performance problems reported several key  factors that contributed to manufacturing problems. These include the  inattention to manufacturing during planning and design, poor planning  for supplier management, and lack of a knowledgeable manufacturing  workforce. Capturing critical manufacturing knowledge during the  planning and design phases before entering production helps to ensure  that a weapon system will work as intended and can be manufactured  efficiently to meet cost, schedule, and quality targets. The programs in our  review often lacked manufacturing knowledge at key decision points,  which led to cost growth and schedule delays. For example, the Joint Air- to-Surface Standoff Missile program—an autonomous, air-to-ground  missile designed to destroy high-value targets—experienced a critical unit- cost breach due to missile reliability problems not being addressed early in  the design phase. Also, the Electromagnetic Aircraft Launch System—a  new catapult technology being developed for the Navy’s newest class of  aircraft carriers—had experienced problems manufacturing compatible  materials, which resulted in cost growth and schedule delays and was the  focus of recent congressional interest. Figure 2 summarizes contributing  factors for manufacturing problems experienced by the four DOD weapon  systems.
As indicated, most of the programs had more than one major problem  related to manufacturing.  These issues illustrate the major problems we  discussed with defense and contractor officials, but do not encompass all  the manufacturing problems experienced by the programs.  For example, recent Air Force study reports that manufacturing and quality assur ance  requirements are not included in the contracts to develop weapon  systems, which could affect the contractor’s approach to manufactu Officials from the Defense Contract Management Agency—a DOD  component that works directly with defense suppliers to ensure that  supplies and services are delivered on time, at projected cost, and meet performance requirements—also reported similar contra could affect contractor performance on manufacturing.  ring.
section_title: Manufacturing Was Overlooked du Development
paragraphs: Each of the four programs we examined did not give manufacturing strong  consideration during the early planning and design phases. Programs we re  moved into production largely without considering manufacturing risks  earlier in the acquisition process, as demonstrated by the experiences of  the Exoatmospheric Kill Vehicle and the H-1 helicopter upgrade program The Exoatmospheric Kill Vehicle was designed to intercept and destroy  high-speed ballistic missile warheads in mid-flight, while the H-1 upgrad program converts the attack helicopter and the utility helicopter to the  AH-1Z and UH-1Y configurations, respectively.  .
The Exoatmospheric Kill Vehicle program was put on an accelerated  development schedule in response to a directive to develop and deploy, the earliest possible date, ballistic missile defense drawing on the best  technologies available. According to the contractor, it bypassed some of  its normal development-review processes to accelerate delivery of the  vehicle, which also resulted in a high acceptance of manufacturing risks  without sufficient identification and management of risk-mitigation p For example, the program went into production without completing  qualification testing. In addition, the contractor continued to incorpo design changes while supplier production was ongoing, resulting in  rework and disruption to the production line. Early lots of kill vehicles  were built manually by engineers in the absence of automated productio processes, which caused dissimilarities among vehicles in the fleet and  will make refurbishments difficult. lans.
For several reasons, the H-1 helicopter upgrade program did not inclu manufacturing in the early phases of planning and also proceeded to  de  production before its design was mature, according to the contracto First, the program underestimated the complexity of updating and  remanufacturing the aircraft without historical drawings. The emphasis  was placed on minimizing development costs and resources were not  available to assess manufacturing challenges early in the redesign process. Furthermore, the program started low-rate production before completing  operational evaluation testing. As a result, the problems uncovered during  testing had to be corrected on aircraft that were on the assembly line.  Also, constant change orders and factory bottlenecks, among other  problems, affected program costs and schedules. The schedule pressure  allowed little opportunity to remedy the manufacturing problems,  resulting in more complicated and expensive fixes. Ultimately the  schedule slowed and the costs increased to the point that the program  abandoned the remanufacturing upgrade and, instead, opted to purchase  newly manufactured aircraft cabins for the UH-1Y configuration.  r.
section_title: Poor Planning Led t Supplier Problems
paragraphs: Inattention to manufacturing during planning and design led to subsequ problems with supplier management in two major defense acquisition  programs we reviewed.  Specifically, the prime contractors did not give adequate attention to managing their suppliers. For example, program officials for the Joint Air-to-Surface Standoff Missile told us that the  responsibility for manufacturing processes and discipline shifted in the  1990s from the government to the defense contractors. The government started to rely on the prime contractor to ensure quality and reliability,  particularly with subtier suppliers. In this case, the program office told  that the prime contractor for the missile program relied on the subtier  suppliers to self-report their capabilities and did not engage in effective  oversight of their work, which led to defective parts. The program office  recently recruited experts in manufacturing to help  the prime contractor  address their supplier problems more effectively.
section_title: Lack of Manufacturing Knowledge Contributed to Problems
paragraphs: Some DOD programs and prime contractors had an inadequate defense  manufacturing workforce—both in terms of numbers and experience—to  effectively manage and oversee manufacturing efforts, which resulted in  schedule delays or cost inefficiencies. The manufacturing workforce  includes occupations such as specialists in quality assurance, business,  manufacturing engineering, industrial engineering, and production control.  In many cases, the programs lacked manufacturing expertise early in  development, which hindered the program’s ability to later manage  manufacturing risks. For example, the contractor for the Electromagnetic  Air Launch System did not have sufficient systems-engineering personnel  involved in the design to help it transition from development to  production. As a result, the program encountered schedule delays and cost  increases. DOD conducted a program assessment review, which led the  program office and contractor to increase systems engineering staff.
For the Exoatmospheric Kill Vehicle program, the contractor’s workforce  and manufacturing processes could not readily undertake the rigors of  production for a space-based capability, part of which must be  manufactured in a clean room environment, and all of which commands  rigorous processes and procedures due to highly technical designs. The  contractor’s hourly assembly personnel were trained to build tactical  missiles on a high-rate production line and were not sufficiently trained in  the quality-control standards required by clean-room manufacturing, such  as carefully controlling foreign-object debris, specially maintaining the  clean room, and using a partner in certain high-level tasks to ensure all  steps are properly followed. These standards were not institutionalized,  and the contractor eventually had to modify its facilities and production  standards to correct the manufacturing problems. The facility had to be  retooled and reconfigured late in development. The contractor also  experienced high turnover in its workforce due to the increasing demands  associated with working in a clean-room environment and working long  hours.
section_title: MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages Manufacturing Risk and Readiness
paragraphs: The Joint Defense Manufacturing Technology Panel working group has  proposed MRLs as new manufacturing readiness criteria that could  improve weapon system outcomes by standardizing the way programs  identify and manage manufacturing risks associated with developing and  fielding advanced weapon systems. MRLs were first introduced to the  defense community in DOD’s 2005 Technology Readiness Assessment  Deskbook as an important activity for science and technology and  acquisition managers to consider. An analysis by the working group shows  that MRLs address many of the manufacturing issues not covered by  DOD’s technical reviews, particularly reviews conducted in the early  phases of acquisition. In their development, comprehensive efforts were  undertaken to design and develop MRLs from DOD as well as industry  resources. For example, the working group formulated MRLs from a  manufacturing knowledge base of defense, industry, and academia to  address two key areas of risk—immature product technologies and  immature manufacturing capability. The working group also designed  MRLs as a structured and disciplined approach for the way manufacturing  risk and readiness is expected to be identified and assessed. The working  group also developed a set of tools that include a deskbook, checklist, and  a website to help managers and users apply MRLs and conduct  assessments. In addition, the Army and Air Force report that their use of  MRLs on pilot programs contributed to substantial cost benefits on a  variety of programs, including major acquisition programs.
subsections: section_title: MRLs Were Developed from Knowledge-Based Resources on Manufacturing
paragraphs: To develop MRLs, the working group conducted comprehensive sessions  with industry participants to ensure the metrics and vocabulary for  assessing manufacturing readiness would be an all-inclusive body of  knowledge. Officials stated that a mature set of manufacturing knowledge  resources already existed but it was scattered and not consistently applied  in a disciplined way that aligned with the DOD acquisition life-cycle  framework. In their formulation, MRLs were developed from an extensive  body of manufacturing knowledge that included, but was not limited to,  the following defense, industry, and academic sources:    DOD Instruction 5000.02, Operation of the Defense Acquisition System  (Dec. 8, 2008),    Navy best-practices manual for using templates on design and    Air Force manufacturing development guide,   military standards and specifications, and    Malcolm Baldrige quality award criteria.
Other standards and technical sources were obtained from the Institute of  Electrical and Electronics Engineers, the International Standards  Organization on quality management systems, automotive industry quality  standards, and the supplier model from the Massachusetts Institute of  Technology.
section_title: Analysis Shows MRLs Address Manufacturing Gaps in DOD’s Technical Reviews
paragraphs: An analysis conducted by the working group shows that MRLs address  many of the manufacturing gaps identified in several of DOD’s technical  reviews that provide program oversight and determine how well  programs are meeting expected goals, particularly the reviews conducted  in the early acquisition phases. According to the working group,  addressing these manufacturing gaps is fundamental to improving the way  programs plan, design, and prepare for manufacturing. For example, the  working group’s analysis shows that DOD’s current systems-engineering  technical review checklist used for preliminary design reviews has only  27 of 759 total questions that deal with core manufacturing-related  questions, whereas the MRL 6 assessment checklist for this juncture has  169 core manufacturing questions. More importantly, the technical review  checklist did not address key manufacturing discipline in the areas of  program management, systems engineering, requirements management,  risk management, and program schedule. Similarly, the technical review  checklist used for critical design reviews has only 22 of 824 total  questions that deal with core manufacturing questions, whereas the MRL 7  assessment checklist for this juncture has 162 core questions. Core  manufacturing disciplines were not addressed in the specific areas of  management metrics, manufacturing planning, requirements management,  system verification, and other areas. Finally, DOD’s technical review  checklist used for production readiness reviews has 194 of 613 total  questions that deal with core manufacturing questions. While the MRL 8  assessment checklist has 14 fewer core questions on manufacturing at this  juncture, the working group stated these core manufacturing questions are  addressed earlier in the acquisition framework, which is reflective of  commercial best practices where such manufacturing topics and discipline  are addressed, in contrast to DOD’s current practice.
section_title: Draft Deskbook Explains MRL Application and Assessments
paragraphs: The draft MRL deskbook is a detailed instructional resource on how to  apply MRLs and conduct assessments of manufacturing risk and readiness,  such as how to structure and apply evaluations to a technology,  component, manufacturing process, weapon system, or subsystem using  the MRL definitions. It also demonstrates how assessments should be  carried out at various phases by the managers of science and technology  projects and technology demonstration projects intending to transition  directly to the acquisition community, as well as acquisition program  managers and the people involved in conducting assessments. According  to the working group, MRLs can not only be used to improve how DOD  manages and communicates manufacturing risk and readiness, but can  also give decision makers and manager’s better visibility into program  risks. For example, a variety of manufacturing status and risk evaluations  have been performed for years as part of defense acquisition programs in a  variety of forms—for example, production readiness reviews,  manufacturing management/production capability reviews, etc. However,  these structured and managed reviews do not use a uniform metric to  measure and communicate manufacturing risk and readiness.
MRLs, when used in combination with technology readiness levels, are  expected to address two key risk areas—immature product technologies  and immature manufacturing capability. The draft deskbook says that it is  common for manufacturing readiness to be paced by technology readiness  or design stability, and that it is not until the product technology and  product design are stable that manufacturing processes will be able to  mature. MRLs can also be used to define manufacturing readiness and risk  at the system or subsystem level. For these reasons, the MRL definitions  were designed to include a target level of technology readiness as a  prerequisite for each level of manufacturing readiness. Figure 3 shows the  relationship of MRLs to system milestones and technology readiness levels  in the defense acquisition life-cycle framework.
section_title: MRL Assessments Provide Basis for Identifying, Planning, and Managing Program Risks
paragraphs: MRL assessments are intended to leverage better manufacturing  knowledge, enabling managers to be aware of problems or risks early in  development, when they are easier to resolve and before significant  investments are made. In turn, these risks can be addressed earlier in the  life cycle when costs are lower. For example, the ability to transition  technology smoothly and efficiently from the laboratories, onto the factory  floor, and into the field is a critical enabler for evolutionary acquisition.
Assessments can be applied to a technology, manufacturing process,  weapon system, or subsystem using the definitions as a standard. As part  of the assessment, a comparison is made between the actual MRLs and the  target MRL levels. The difference between the two identifies the risks and  forms the basis for assisting managers to develop a plan—called a  manufacturing maturation plan—to remove or reduce them. Risks should  be identified throughout the life cycle and, when targets are not met, the  plan updated to ensure the appropriate MRL will be achieved at the next  decision point. The manufacturing maturation plan identifies  manufacturing risks and provides a plan for mitigating each risk area  throughout the duration of the technology or product-development  program. The draft MRL deskbook says every assessment of  manufacturing readiness should have an associated plan for areas where  the MRL has not achieved its target level. The deskbook requires a  manufacturing maturation plan to include the most essential items in  planning for the maturity of an element of assessment that is below its  target MRL. These include a statement of the problem that describes areas  where manufacturing readiness falls short of the target MRLs, including  key factors and driving issues, solution options and consequences of each  option, and a maturation plan with a schedule and funding breakout. Other  information should include the status of funding to execute the  manufacturing plan and specific actions to be taken and by whom, and the  MRL to be achieved and when it will be achieved.
section_title: MRL Pilot Programs Show Positive Benefits
paragraphs: Army and Air Force programs have pilot-tested MRLs on science and  technology and some major acquisition programs in an effort to increase  the manufacturing readiness and maturity to higher levels appropriate to  the phase of development. Both services performed MRL assessments on  selected pilot programs to address manufacturing risks and assess  technology transition. The Army reports numerous benefits from the use  of MRLs such as manufacturing efficiencies, improved labor utilization,  and cost benefits. Similarly, the Air Force has used MRLs to manage its  manufacturing risks associated with new technologies, yielding tangible  benefits. While MRLs cannot take full credit for all benefits derived in the  pilot programs, officials noted they are a good way to manage, mitigate,  and communicate—between science and technology, acquisition, the user,  and the system developer—readiness and risks early and throughout the  acquisition process to avoid major consequences from manufacturing- related problems. These programs provide insight on how the acquisition  community can utilize MRLs within weapon system programs.
subsections: section_title: Army
paragraphs: In 2004, the Army’s Aviation and Missile Research, Development and  Engineering Center began applying MRLs to various technologies in  concept development, including those technologies transitioning to  engineering and manufacturing development. Officials stated that without  cost and manufacturing readiness planning, science and technology  programs face certain barriers to transition, resulting in: (1) high unit  production cost caused by a focus on technology without regard to  affordability; and (2) manufacturing problems caused by design  complexity resulting in a technology that is not feasible to manufacture.  For example, the Army has applied MRLs to many programs, including  warfighter-protection materials, Micro-Electro-Mechanical Systems,  embedded sensors, and helicopter cabin structures. The warfighter- protection program—the next generation of helmets and body gear— reported that it was able to reduce scrap by 60 percent and reduced touch  labor by 20 to 40 percent. On programs where cost benefits could be  roughly calculated, the Army believes that MRLs, among other  improvement initiatives, contributed to the $426 million in benefits on  seven programs. MRLs were also used as a metric in the Technology  Transition Agreement to communicate manufacturing maturity and  facilitate a smooth transition to the acquisition community.
section_title: Air Force
paragraphs: Air Force officials we met with discussed using MRLs to assess and  identify gaps and understand risks in manufacturing maturity that would  delay technology transition into an advanced systems development  program or a fielded system upgrade. The Air Force has conducted several  MRL assessments on advanced technology demonstrations and major  defense acquisition programs, including the MQ-9 Reaper Unmanned  Aircraft, Joint Strike Fighter, Advance Medium-Range Air-to-Air Missile,  X-band thin radar array, and Sensor Hardening for Tactical Systems.  Officials reported that the use of MRLs have contributed millions of  dollars in cost avoidance, increased production rates, and has accelerated  technology transition. For example, the Air Force reported realizing $65  million in savings by addressing problems with a costly manual drilling  process. MRLs were used to raise new drilling technology from MRL 4 to  MRL 9, achieving a unit-cost savings of $17,000 per aircraft from reduced  tooling, manpower, floor space usage, and time.
Because of MRL assessment’s success on advanced technology programs,  the Assistant Secretary of the Air Force for Acquisition directed the  program office to perform MRL assessments on key MQ-9 Reaper  manufacturing processes and technologies. The MQ-9 Reaper is an  unmanned aerial vehicle designed to provide a ground attack capability  during reconnaissance and surveillance missions. Officials stated that the  MRL assessment results have (1) identified five areas that needed review  prior to a milestone C production decision; (2) identified two risks to full- rate production—mitigations are in progress; and (3) provided evidence to  support the contractor’s ability to meet the production goal of two aircraft  per month. To ensure that manufacturing requirements are enforced,  officials have developed policy for programs managers to assess  manufacturing readiness at key decision points. To support that policy, the  Air Force has developed training for integrated product teams to execute  the manufacturing readiness assessments. Also in August 2009, the Air  Force Institute of Technology established a Manufacturing Readiness  Assessment course to provide training for the assessments within the Air  Force and is currently open to all services and industry.
section_title: DOD’s Proposed MRLs Embody Many Best Practices of Leading Commercial Firms
paragraphs: To successfully develop and manufacture their products, the commercial  firms we visited used a disciplined, gated process that emphasized  manufacturing criteria early and throughout the product’s development.  To measure manufacturing maturity, these firms developed processes that  give manufacturing readiness and producibility primary importance  throughout the product-development process, focusing on producing a  product, not developing a technology. The goal is business profitability,  and manufacturing maturity is important to this process from the earliest  stages.
The best practices they employed were focused on gathering a sufficient  amount of knowledge about their products’ producibility in order to lower  manufacturing risks and included stringent manufacturing readiness  criteria—to measure whether the product was mature enough to move  forward in its development. In most respects, these criteria are similar to  DOD’s proposed MRLs. For example, as with MRLs, commercial firms  assess producibility at each gate using clearly defined manufacturing  readiness criteria,  gain knowledge about manufacturing early,    demonstrate manufacturing processes in a production-relevant  emphasize the importance of effective supply-chain management.
Essentially, commercial firms emphasize these criteria in order to  maximize their understanding of manufacturing issues, to mitigate  manufacturing risks that could affect business profitability or schedule  goals for getting the product to market. DOD’s MRLs were designed to  mitigate similar manufacturing risks. However, the difference is that the  commercial firms we visited required that their manufacturing processes  be in control prior to low-rate production, whereas DOD’s proposed MRL  criteria do not require as early control of the manufacturing process.
subsections: section_title: DOD’s MRLs Are Similar to Manufacturing Criteria Used by Leading Firms
paragraphs: Leading commercial firms use manufacturing readiness criteria, similar to  DOD’s MRLs, to assess the producibility of a system, gathering knowledge  about the producibility of a product and the maturity of the manufacturing  process. These criteria are applied early, even before a product formally  enters into development, to identify and manage manufacturing risks and  gaps. Additional manufacturing readiness criteria are applied through all  the stages of a product’s development and production until the product is  ready for commercial release. The firms we visited used manufacturing  readiness criteria to measure both the readiness of the product or material  to enter into development and to proceed through the necessary gates.  Table 3 below shows examples of manufacturing readiness criteria that  are common to both the MRLs and the commercial criteria, to illustrate  their similarities. Both emphasized identifying risks and developing plans  to mitigate these risks, setting realistic cost goals, and proving out  manufacturing processes, material, and products.
section_title: Best Practice: Commercial Companies Emphasize Manufacturing Criteria Early and at Every Stage of the Product-Development Life Cycle
paragraphs: Each commercial firm we visited developed a disciplined framework for  product development that assessed producibility at each gate using clearly  defined manufacturing-maturity criteria that are similar in many respects  to DOD’s MRLs. These include assessments of all aspects of  manufacturing technology and risk, supply-chain issues, production  facilities and tooling, and materials. Throughout the product-development  life cycle, these criteria were applied to determine entry or exit into the  next phase and led to informed decisions about whether the product was  ready to move forward in its development. Manufacturing risks—such as  those found in new manufacturing technologies or production facilities,  new or revolutionary materials or supply-chain issues—were assessed at  each step. Deliverables, including risk-identification and mitigation plans,  manufacturing plans, and funding and resource needs, were required at  each gate in order to progress to the next product-development gate.  Targets were developed for each gate, including cost, schedule, and yield  goals, and the product team was responsible for either meeting these  targets or having risk-mitigation plans in place if the targets had not been  met.
GE Aviation exemplifies this disciplined process, using a highly structured  gated process with detailed checklists for entry and exit into each phase.  Like DOD’s MRLs, these checklists contain increasingly detailed criteria— as they move from product start to production—for evaluating  manufacturing technologies, cost drivers, materials, and supply-chain  issues. Structured teams are brought together, tools are identified for  execution and control of the process, and scheduled reviews are  conducted with defined deliverables and checklists for each milestone. At  each milestone, a vigorous review of the plans for the product’s  development and manufacturing and risk-reduction efforts highlights  issues before they become problems. The firm’s goal is to have mature  processes by production. To achieve this, it considers manufacturing  readiness throughout. Each project’s team is cross-functional and includes  senior management, mid-management and the project team. This robust  review process leverages expertise across GE Aviation, reduces risk, and  highlights issues before they become problems.
As with all the commercial firms we visited, GE Aviation requires strong  management involvement at each gate, along with decision reviews to  determine if enough knowledge is available and risk-mitigation plans are  in place to proceed or if actions to address and mitigate manufacturing  risks can show a viable way forward. This allows management to resolve  problems rather than pass them on to the next phase. At project start,  which corresponds to MRL 4, the senior leadership team and product  leadership team generate the product idea and assess the need for the  project. They provide linkage between the business strategy and the  project and develop the high-level project strategy. They identify any new  product material or manufacturing processes and begin to develop a risk- reduction strategy for these issues. By the time the product enters the  preliminary design phase, senior leadership and project teams agree on the  approach to the project. At this time, product directors must have a  manufacturing plan in place in order to identify how they are going to  achieve manufacturing readiness. Technical risks are identified in the  manufacturing plan, as well as risk-abatement strategies for materials and  manufacturing processes and supply-chain risks. The plan has to show  how issues will be successfully addressed by the detailed design phase,  when leadership, the project team, and customers agree on the product to  be delivered. If agreement is reached, they freeze the project plan and a  decision is made to fund or terminate the project.
subsections: section_title: Multidisciplinary Team / Manufacturing Experts
paragraphs: In the commercial firms we visited, product-development teams were  multidisciplinary, generally including management, manufacturing, quality,  finance, suppliers, and engineering, with necessary skills available to  assess manufacturing readiness. Leading firms recognize the value of  having a knowledgeable, well-trained, and skilled manufacturing  engineering workforce involved in these multidisciplinary teams from the  beginning and throughout the process. When Honeywell reorganized its  aerospace business in 2005, it created an advanced manufacturing  engineering organization to focus on manufacturing concerns in the  earliest phases of new product-development programs. This organization  consists of engineers to support various manufacturing disciplines in  Honeywell. An important part of this advanced engineering organization is  its technology group, which consists of a select number of technology  fellows with extensive expertise in key manufacturing disciplines that  touch nearly all the products Honeywell produces. Honeywell retains  highly skilled manufacturing expertise through this program and uses  these experienced and knowledgeable manufacturing engineers to oversee  each project’s manufacturing assessments.
section_title: Maturing Technology and Manufacturing Processes
paragraphs: Commercial firms focus on maturing and validating technology and  manufacturing processes before these are associated with a product and  before entry into the gated process. They keep invention and unproven  technologies in the technology base until their producibility at the scale  needed can be proven. As an example, GE Healthcare’s Gemstone  scintillator underwent years of laboratory development on a small scale  until GE Healthcare was satisfied that this material was ready to be used  on its computed tomography (CT) scanners. Scintillators work by  converting the X-rays in the CT scanner into visible light. GE Healthcare  had been manufacturing its own scintillators since the late 1980s, but it  needed an improved one that worked faster, for better clarity of vision and  to reduce the amount of exposure to radiation. In 2001, the firm began  basic composition development at the laboratory scale and narrowed  down the alternatives to find the material with the best properties for this  use. Even at this early stage, several years before the material would enter  into GE Healthcare’s gated process, there was early engagement by the  chemists with the manufacturing side. Before they decided on a solution, a  determination was made that it could produce them with sufficient yield  and quality: even if a material had the best optical qualities, it had to  balance this with its producibility. GE Healthcare tested thousands of  alternatives to determine what could meet its technical requirements and  be producible in the quantities needed. The firm narrowed it down to a  garnet-based, rare-earth minerals composite, and began producing it in  small but increasing quantities. After narrowing the field to this garnet- based compound, GE Healthcare began to determine its suppliers and  what equipment was needed. The firm then began building its first pilot  plant to produce the material and the scintillators, 2 years before the  scintillator entered the firm’s gated process. Figure 4 shows a photo of a  CT scanner that uses the scintillator technology.
section_title: Best Practice: Commercial Firms Have Adopted DOD’s MRLs or Are Employing Similar Criteria in Their Product- Development Process
paragraphs: Because leading commercial firms focus on producibility as a key element  to successfully develop products, they use rigorous analysis methods to  assess producibility and to identify and manage manufacturing risks and  gaps. They apply these methods and tools early and throughout product  development and use them to manage their product development on a  daily basis. This commercial approach is a process in which quality is  designed into a product and manufacturing processes are brought into  statistical control to reduce defects, in contrast to practices employed by  many defense contractors where problems are identified and corrected  after a product is produced.
Some firms were familiar with the DOD MRL proposal and had taken st to use the concepts at their own companies. Honeywell, for example,  determined that early decisions were responsible for many production  issues and so they developed analytical tools and models that support  evaluations of manufacturing and risk throughout the product- development life cycle. In 2005, Honeywell engineers began looking for a  way to measure manufacturing readiness and producibility, since they  realized that early program decisions were driving many production iss and that by the time a product entered engineering and manufacturing  development, it was too late to efficiently affect these issues. Some of  these issues include cost overruns, quality problems, low-yield issues,  service and maintainability inefficiencies, and supply-chain problems.
The output of this tool is an MRL assessment score that can identify gaps  or risks. For example, spreadsheets show the MRL scoring at a glance for  each of the elements evaluated, pinpointing the gaps; risk worksheets to  quantify the risks; and action plans to close the gaps and mitigate these  risks. It links to the firm’s gated process, providing entry and exit criteria  and feedback on how to meet these criteria. The important information  obtained is not necessarily what MRL level the item is at currently, but  rather the robustness of the gap-closure plan to get to the desired level for  the next gate. The application of the MRL tool helps identify what these  key gaps are and what steps are required to close them.
The three enabling producibility tools that provide support for this  assessment and early input on the producibility risks are: a Design for  Manufacturing Model, a Product Complexity Model and a Yield Prediction  Model:    Manufacturing Complexity Model: This model identifies the design  features that are driving manufacturing complexity into the design and  enables scenarios to be evaluated to see what actions can be taken to  simplify the design. Higher-complexity designs generally cost more and  are higher risk, so the goal is to identify alternative design solutions  that minimize complexity, but still meet all the performance  requirements.
Yield Prediction Model: Honeywell has also developed yield prediction  models based on statistical principles that correlate opportunities for  defects in a design to established process capability benchmarks. This  approach is used to predict yield during early design activities based  on knowledge of the manufacturing processes used and the complexity  of the design.
Design for Manufacturing Scorecard analysis: The third Honeywell- developed tool is a design for manufacturing scorecard, which  quantifies how well the design adheres to recommended best  practices. The goal of using the tool is to provide feedback to the  designers so that they see how their design decisions directly affect  producibility and help pinpoint improvement areas early in the  process.
Honeywell then conducts an MRL workshop, with a team led by an  engineer from its Advanced Manufacturing Engineering group that  includes the program manager and various subject-matter experts.  This  team reviews the tools and the MRL criteria to gain consensus on ratings  for each category. Honeywell’s Manufacturing Maturity Model, with input  from these enabling tools, is used to develop an MRL score for the  product. These assessments provide early producibility evaluations  essential to mitigating design-driven risks. Since many producibility issues  are driven by early design architecture decisions, these tools provide a  way to analyze these decisions early and make the necessary performance  and producibility trades through “virtual prototyping” long before actual  hardware is built. The MRL score provides the necessary framework to ask  the questions that such an analysis needs to answer.
After the MRL assessment is complete and the MRL scores and risk- mitigation plans are approved, the MRL analysis and risk mitigations are  incorporated into the daily schedule of the program office. The office  continually monitors the MRL levels, updating them and working toward  its risk mitigation goals.
section_title: Best Practice: Leading Firms Prove Out Manufacturing Tooling, Equipment, and Processes before Entry into Production
paragraphs: Companies we visited spent years prior to production developing and  proving out their manufacturing processes, including building test articles  on pilot production facilities to perfect these processes. This allowed them  to perfect and validate these processes, eliminate waste and scale up  gradually to the required manufacturing level. They reduce errors and  inefficiencies with the purpose of retiring manufacturing risks.
GE Aviation officials told us that certain advanced manufacturing  technologies achieve significant cost savings by getting the costs lower  earlier in the process and decreasing cycle time for faster implementation.  An example of manufacturing techniques or processes that have made a  big difference in costs, accuracy, and reliability include processes for  drilling small shaped holes for turbine airfoils.
GE Aviation’s Turbine Airfoils Lean Lab provides a mock-up of a  production facility or process, where such technologies and production  processes can be tested to eliminate waste, scrap, and excess steps. They  focus on one part family or process, such as the turbine airfoil shaped-hole  manufacturing. The turbine airfoil is a part of the jet engine that generates  power—it extracts horsepower from the high-temperature, high-speed  combusted gasses. Turbine airfoil blades require hundreds of cooling  holes that help maintain part integrity at elevated operating temperatures.  Traditionally, round holes were used, but the technology has evolved to  compound-angle-shaped holes, which improve cooling effectiveness and  reduce engine stress. These type of holes cannot be economically  produced by traditional methods and require improved manufacturing  techniques. Advanced laser drilling was determined to be feasible, and GE  Aviation decided to initiate the program through the Lean Lab to ensure  manufacturing readiness of the process.
GE Aviation officials compared their processes in this case to DOD’s  MRLs. Prior to entering their gated process, they began making  investments in potential technologies, including tooling (MRL 1-3). As the  gated process began, risks were identified and risk-abatement plans were  put in place (MRL 4). GE Aviation then set up the Lean Lab to test the way  the airfoil would actually be built. New processes were introduced that  included new laser methods for hole drilling, improved robotic technology,  machining, and grinding (MRL 5-6). The managers then ran the pilot  production line for some time to manufacture these airfoils using actual  production operators to be confident that the process would translate to  the production line. Adjustments were made to improve efficiency and  retested on the line until they were satisfied that they had worked out the  best procedures. GE had tooling-design experts on the team at the Lean  Lab to provide rapid part and tool manufacturing. Processes were brought  into statistical control in order to take the complexity out of  manufacturing, simplify the process, and reduce waste (MRL 7-8). They  then dismantled the production line at the Lean Lab, took it to the  manufacturing facility, and set it up exactly the same, with no variations  allowed (MRL 9). This seamless introduction of the new manufacturing  technology and the lean principles developed in the lab are expected to  save many millions of dollars across GE Aviation, on production of this  part family alone. Figure 6 shows a photo of GE Aviation’s Lean Lab setup.
GE Healthcare provides another example of proving out manufacturing  processes prior to production in their development of the Gemstone  scintillator for use on their CT Scanners. In 2003, the technology for this  transitioned into the firm’s formal gated process or product start-up, and it  began a detailed and extensive development of the manufacturing process.  The firm built a pilot plant for this purpose and began manufacturing the  composite in increasing amounts. In this first pilot plant, it was able to  process the materials in increased quantities from what it produced in the  lab. GE Healthcare verified that it had the right technologies to minimize  manufacturing risks. In the laboratory environment, the firm had already  answered the question “Can this composite be made with the desired  properties?” and now asked “Can it be made with sufficient yield and  quality to be manufactured in the desired amounts?” This early  engagement with manufacturing enabled the firm to develop the process  and reduce errors and inefficiencies with the purpose of reducing  manufacturing risks.
GE Healthcare then built a second pilot production plant that further  increased the amount produced above that of the first pilot plant. The firm  continued its focus on gaining knowledge early, but on a larger scale:  building the pilot plants was important to perfecting the process and  gaining knowledge about the material’s producibility. At this stage, which  coincides with MRL 8, it eliminated most of the technical risks involved in  manufacturing the material. The firm then began to build its full-scale  facility, which was ready 18 months before product launch.
When the full-scale production facility was completed, further scale-up of  the material’s manufacturing became the focus. Changes to the design  were made as needed to facilitate this. Any remaining manufacturing risks  were eliminated prior to entry into the next stage, the product-validation  stage. The Food and Drug Administration requires validation of finished  medical devices. GE Healthcare told us that this means that all the  equipment, processes, procedures, and factory workers are the same as  will be used in actual production. Through use of the pilot plants to  perfect the manufacturing of the scintillator material, GE Healthcare was  able to produce production-representative material to satisfy this  requirement.
section_title: Best Practice: Commercial Firms Work Closely with Suppliers, Who Must Meet High Quality Standards for Parts and Supplies
paragraphs: Commercial firms focus on developing strong relationships with their  suppliers to ensure quality parts are provided in a timely manner. This  begins with rigorous supplier-selection criteria to create a strong supplier  base to provide quality parts. Similarly, DOD’s MRL supply-chain thread  focuses on supplier capability throughout the acquisition life cycle, from  as early as pre–milestone A (MRL 3), where initial assessment of the  supply chain begins, through MRL 5, where supply-chain sources have  been identified, and continuing to MRL 8, where the supply-chain should  be stable and adequate to support low-rate production. Commercial firms  generally have long-term relationships with these suppliers and can  identify the supplier that is the best source of material or parts early, well  before production begins. Leading commercial firms apply the same  standards to these suppliers as they apply to their own manufacturing  processes, such as ISO 9000 or other quality standards. Throughout  product development and production, they establish effective  communications with their suppliers so they can continually assess their  performance. These firms work closely with their suppliers to retain these  beneficial relationships, providing training where necessary and assistance  if manufacturing problems arise.
GE Healthcare suppliers have to be validated before production begins,  but qualifying them starts in the design phase. Suppliers are expected to  meet the ISO 9000 standards and the Food and Drug Administration’s  medical devices standards, but GE Healthcare’s own standards are more  stringent that those. The supplier-qualification process ensures that  suppliers meet GE Healthcare’s requirements, have a quality system that  provides the appropriate controls for the part provided and meet  regulations and requirements of multiple agencies, such as the Food and  Drug Administration. Once a supplier is qualified, it becomes an approved  supplier.
GE Healthcare also audits most of its suppliers and looks for issues such  as lapsed ISO 9000 certification or a failed review. If it finds these things,  GE Healthcare will ask the supplier for a plan to correct the deficiency and  reaudit the supplier. GE Healthcare does annual risk assessments on the  suppliers, based on data gathered during these audits, with sole-source or  single-source suppliers being a high risk. If a supplier falls out of qualified  status, GE Healthcare will do more frequent assessments. It constantly  monitors the suppliers for quality. It helps the supplier get to the quality  needed, but quality goals must be met.
Siemens is a global company that employs about 70,000 people in the  United States. We visited Siemens Mobility Division, which builds light rail  cars for public transit. Siemens places special emphasis on its supplier  relationships, since it knows its suppliers can contract to other rail-car  builders, as there is competition for suppliers in this market. If it has a  good relationship with its suppliers, it can continue to benefit from the  relationships with high-quality suppliers. Once it qualifies a supplier, it  takes the responsibility for keeping the supplier qualified, providing  technical assistance if necessary to keep the supplier in its pipeline. Even  as early as the bid phase of the contract, Siemens knows who it will need  as suppliers and if any particular supplier is new or challenged in some  respect.
Siemens applies a three-step supplier-qualification process to its suppliers.  This starts with a supplier self-assessment. The firm’s supplier- qualification personnel then visit the supplier’s plant and evaluate the  supplier on the same self-assessment form, to determine if the supplier  will make it to the vendor-qualification list. Once a supplier is on the  approved vendor-qualification list, Siemens does risk ratings for these  vendors to be sure it can keep them on the qualified-vendor list.  The firm  updates these assessments if the vendor situation changes, rating the  vendor at low risk if it is fully qualified and working with it if some aspects  are not qualified. Siemens takes responsibility for keeping the approved  suppliers qualified, since finding and qualifying new vendors can be time- consuming and risky. It tries not to overload any one supplier, because  some of their suppliers are small or specialty operations, so it keeps a pool  of qualified suppliers for as many parts or materials as it can.
section_title: Commercial Firms Require That Manufacturing Processes Be in Control Earlier Than DOD’s MRLs
paragraphs: Although the firms we visited used manufacturing readiness criteria  similar to DOD's proposed MRLs, one important difference we observed is  that the commercial best practice is to have manufacturing processes in  control prior to the production decision, while DOD's MRLs require  manufacturing processes and procedures to be established and controlled  during MRL 9, which occurs after the milestone C production decision,  which authorizes a program to enter low-rate initial production, or  equivalent. Although DOD’s MRLs incorporate many of the commercial  manufacturing best practices into their manufacturing design and  implementation criteria, the process controls criteria would be met too  late in the process to achieve their full effect. DOD’s MRL matrix states  that low-rate production yield and rate targets should be achieved at MRL  9, after the production decision has been made. The commercial firms we  talked to emphasized that production processes must be in control before  this decision is made. They realize that they are unable to make  predictions about production performance until the process is stable and  defects are predictable. Not achieving process control could result in low  quality, extensive rework and waste, and not meeting cost and schedule  targets. Firms established pilot lines to prove out production material,  processes, and tooling, and worked to get processes under control before  the system could move from the pilot line to production line. Figure 7  shows a depiction of the commercial manufacturing process approach.
The companies we visited used various approaches to build process  capability and provide timely information on whether manufactured  components, subsystems, or systems meet design specification. For  example, GE Aviation uses a statistical measurement, called Z sigma level,  to determine whether its processes have been brought under control or if  variations in its manufacturing process could affect the quality of the  product. The product is not moved into production until the firm is  satisfied that these processes are in control. Similarly, GE Healthcare’s  milestone process requires that a set of quality targets are part of the  program and that those quality targets are met. Measures of process  control vary from company to company, such as using yield or scrap and  rework rates or sigma levels, but each looks carefully at those measures to  ensure they carried no product-quality risk and uses this information to  determine if the product is ready to be manufactured.
section_title: Two Successful DOD Programs Used Criteria Similar to Commercial Firms
paragraphs: Two DOD programs, the Army’s Lakota aircraft and the Missile Defense  Agency’s Standard Missile 3 Block 1A, that had successful manufacturing  outcomes employed some of the same practices as leading commercial  firms. Both used a type of manufacturing readiness criteria to evaluate  whether the programs were ready to enter into production and both  programs focused on manufacturability as a key indicator of program  success, using well-developed technology and a conservative approach in  design and development.
The Lakota aircraft, a light utility helicopter that conducts noncombat  missions, was a mature aircraft design when the Army entered into the  contract with the European Aeronautic Defence and Space Company to  purchase this commercially available helicopter. The program shows how  careful attention to manufacturing readiness can reduce program risks.  According to program office officials, the contractor was chosen in part  because of its manufacturing track record, and it completed extensive  planning, both internally and with its supplier base, to ensure on-time and  reliable deliveries. Production planning and preparation were  accomplished, including assessments of the manufacturing processes,  capabilities, and facilities. These assessments determined that the program  was low risk and ready for full-rate production. The Lakota is currently in  full-rate production and has met its cost and schedule targets.
The Standard Missile 3 is a ship-based, antiballistic missile used by the  Aegis ballistic missile defense system. Similar to the Lakota, the system  met its cost and schedule goals by using an incremental, low-risk  approach. Like the commercial firms we visited, the program built  knowledge through the use of a type of manufacturing readiness criteria,  which allowed the early identification of risk and implementation of  mitigation strategies. The Standard Missile 3 Block IA was also on target  for manufacturing cost and schedule and reported a lower cost per unit  than was originally estimated on its production buys. As in the successful  commercial firms we visited, manufacturing issues were considered very  early in the design phase, leading to minimal changes in the program from  flight test to production.
section_title: MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing Workforce Concerns
paragraphs: While acceptance of MRLs is growing within DOD and the defense  industry, the services’ leadership appears to be resistant, and adoption  efforts have been slow. For example, obtaining agreement on a policy that  would institutionalize MRLs defensewide has proven difficult. Concerns  raised by the military-service policymakers have centered on when and  how the MRL assessments would be used. Officials responsible for the  draft policy have promoted MRLs as an initiative that can address the  manufacturing element in the design and production of weapon systems,  citing commercial best practices that employ similar methods, and  benefits derived from pilot programs. While extensive efforts have been  made to promote the benefits of MRLs in support of a revised draft policy,  it has taken nearly 2 years to allay concerns and it has not yet been  approved. DOD is likely to face serious challenges even if an agreement is  reached to approve the policy, however, because the number of DOD’s  production and manufacturing career-field employees has diminished,  particularly within the Air Force. Although the services are at the  beginning stages of revitalizing their production and manufacturing  workforce, DOD currently does not have adequate in-house expertise with  the requisite knowledge to assess manufacturing throughout DOD.  Essentially, the military services and Defense Contract Management  Agency have identified knowledge and manpower gaps in their  manufacturing workforce and believe that any initiative deploying MRLs  defensewide could be hampered as a result.
subsections: section_title: Draft Policy to Institutionalize MRLs Has Proven Difficult, but the DOD Community Is Starting to See Its Value
paragraphs: While acceptance of MRLs is growing within DOD and the defense  industry, the Army’s, Navy’s, and Air Force’s leadership appears to be  resistant and adoption efforts have been slow. For example, a July 2008  draft MRL policy memorandum garnered disagreement among the military- service policymakers. The military services’ leadership agreed that MRLs  provide value in the early acquisition phases but disagreed with the  policy’s intent to formalize the process. For example, the MRL policy  memorandum stated that on the basis of analyses by GAO and the Defense  Science Board—as well as positive results on two Air Force pilot  programs—that acquisition category I programs be assessed using the  MRL scale. In particular, the draft policy included provisions that would  require programs at milestone B to be assessed at MRL 6 or beyond for all  critical technologies; programs at milestone C to be assessed at MRL 8 for  all critical technologies; procedures to be coordinated for including  assessments of manufacturing readiness in addition to technology  readiness assessments at milestone B and C; and incorporation of  guidance into training materials and guidebooks on best practices for  addressing manufacturing from the earliest stages of development through  production and sustainment.
In response to the draft policy, each of the military services issued  memorandums in July 2008 to the Under Secretary of Defense  (Acquisition, Technology and Logistics) or the Director, Defense Research  and Engineering, stating they support MRLs and their use earlier in the  acquisition process but they saw limited value in doing formal assessments  prior to milestone C. In general, the services had concerns on when and  how MRL assessments would be used. More specifically, their concerns  included the following:    Evaluation results that could be used as the basis for go / no go  decisions.
A growing number of assessments being levied on acquisition  programs.
Resources required to prove out multiple production lines in a  competitive prototyping environment during the technology- development phase.
Since 2008, officials responsible for the draft policy memorandum have  been working to address concerns raised by the services. According to the  working group, most concerns pointed to a need to clarify how the  information is intended to be used by decision makers at key milestones,  particularly at the earlier milestones. According to the working group  officials we interviewed, the intent is to inform decision makers with  critical information—such as manufacturing risk and readiness measures,  as appropriate to the phase of acquisition—so that knowledge-based  decisions can be made earlier in the process to influence better outcomes  in terms of cost and schedule in the later acquisition phases. Moreover,  they cite that similar methods are employed by leading commercial firms  as a best practice, plus the fact that MRL pilot programs have already  demonstrated significant benefits. The revised MRL draft policy has not  yet been approved. Officials familiar with the status of the draft policy  stated that the leadership at one of the military services is still opposed to  the idea of standardizing MRLs across DOD, and efforts to get approval  have not yet occurred within the Office of the Director, Defense Research  and Engineering.
DOD experienced similar problems introducing technology readiness  levels. There was opposition to the use of technology readiness levels, but  they became a standard for programs to follow, and the standard that  technologies should be demonstrated in a relevant environment became a  statutory requirement for all major acquisition programs seeking to enter  system development. Programs report benefits from using technology  readiness levels.
Some officials believe that MRLs could significantly reduce cost growth.  For example, the Army and Air Force have reported MRLs were a factor  that contributed to benefits of hundreds of millions of dollars in reduced  program costs, improved schedule, and better performance of products.
section_title: MRL Acceptance Is Growing within DOD and Defense Industry
paragraphs: A number of Army, Air Force, and Missile Defense Agency programs—as  well as defense contractors—have embraced MRLs as the method for  assessing manufacturing maturity, risk, and readiness. For example, some  Army commands have opted to use them on their science and technology  efforts that have manufacturing elements, and have developed a formal  process for identifying them. Similarly, two of three Air Force product  centers under the materiel command—the Aeronautical Systems Center  and the Air Armament Center—have recently issued local policy that  mandate the use of MRLs. For example, in a policy memorandum by the  Aeronautical Systems Center, dated October 13, 2009, all programs are  now required to have manufacturing readiness assessments using MRLs,  prior to each major milestone review. The memorandum acknowledged  that the transition to production has historically been challenging for  many programs and that manufacturing assessments are a key tool to  ensure that programs are ready to begin production. The Missile Defense  Agency has included MRLs as part of their assessment criteria. In addition,  senior missile defense manufacturing personnel have developed and  conducted training on how to conduct these assessments.
Similarly, a number of defense contractors have implemented MRLs as a  discipline for identifying, managing, and communicating manufacturing  risk and readiness. These contractors report a number of benefits using  the MRLs, including reductions in program costs and improved production  schedule. For example, in 2006, Raytheon participated in pilot MRL  program assessments involving the Advanced Medium-Range Air-to-Air  Missile and a portfolio of other programs and concluded the approach  makes good business sense to lower risk. Raytheon claimed cost  reductions of 30 percent or more could be achieved by using MRLs.  Raytheon officials state that the combination of technology and  manufacturing assessment processes changes the culture by driving a  collaborative partnership between programs, design, and manufacturing  engineering earlier in the product-development life cycle where maturity  efforts can have the greatest effect on improving program affordability and  predictability. As a result, Raytheon is deploying MRLs as a standard  across the organization. Lockheed Martin is exploring ways to integrate  MRLs within its existing review processes. As previously discussed,  Honeywell adopted MRLs for use on both its defense and commercial  products, and developed several models as an analysis-based approach to  quantify their producibility risks.
section_title: Manufacturing Workforce Knowledge and Manpower Gaps May Impede Implementation of MRLs
paragraphs: The services are in the beginning stages of revitalizing their manufacturing  workforce, largely in response to a February 2006 Defense Science Board  task force report on “The Manufacturing Technology Program: A Key to  Affordably Equipping the Future Force.” The report acknowledged that  both the manufacturing expertise in the workforce and program funding  have declined, thus eliminating much of the engineering and  manufacturing talent across DOD and the industrial base. The report  concluded that what was once a promising career field in the military  services—with promotion paths, training, and professional development— has been systematically eliminated over the past few decades. Table 4  shows the decrease in the manufacturing career field across DOD from  2001 to 2007.
As indicated, DOD’s manufacturing career workforce trends show an  overall decline, with the Army and Air Force having had the biggest  declines at 14 percent and 37 percent, respectively. According to a DCMA  official, the agency experienced about a 30 percent decrease during the  same timeframe. An Army official responsible for workforce planning  activities noted, however, there are no positions designated specifically for  manufacturing, which make it difficult to determine the true career  workforce numbers in this category. Fewer experts mean that fewer  people at both the working level and in leadership positions understand  the processes involved in developing and manufacturing defense systems  and their importance in producing high-quality and reliable systems.  Further, fewer people are capable of conducting production-readiness  reviews, evaluating industry’s work on programs, and staying abreast of  industry research and development. According to a recent study, of major  concern is that recent estimates show 30 percent of the civilian  manufacturing workforce—classified as production, quality, and  manufacturing—are eligible for full retirement, and approximately 26  percent will become eligible for full retirement over the next 4 years. This  means DOD will soon have an exodus of its manufacturing workforce and,  accordingly, must plan for this eventuality.
Although the services are at the beginning stages of revitalizing their  production and manufacturing workforce, program officials believe they  currently do not have the in-house expertise with the requisite knowledge  to assess manufacturing, if MRLs were to be mandated and deployed  across DOD. For example, in interviews with career planning officials at  the military services, most report that they have workforce challenges in  manufacturing knowledge gaps or insufficient number of personnel to  conduct the work, or both. The Defense Contract Management Agency  reported similar manufacturing knowledge gaps due to a lack of focus in  this area, but it now has new leadership in place and is establishing plans  to address these deficiencies. Essentially, these knowledge deficiencies  affect many areas, such as policy support for programs, the ability to  develop an effective strategic plan and investment strategy for  manufacturing technology, the ability to implement MRLs and conduct  assessments, and the ability to effectively and affordably acquire high- quality weapon systems.
section_title: Conclusions
paragraphs: MRLs, resourced and used effectively, offer the potential for DOD to  achieve substantial savings and efficiencies in developing and acquiring  weapon systems. MRLs have been shown to work in reducing the cost and  time for developing technologies and producing systems. Moreover, they  have been shown to work on individual programs, and some Army  commands and Air Force centers have adopted them. They are consistent  with commercial best practices and have even been adopted by some  defense firms. Yet, they have not been adopted DOD-wide. MRLs are being  met with resistance similar to that experienced by technology readiness  levels when they were first introduced. However, technology readiness  levels are now widely accepted and used across DOD.
While MRLs represent a common body of knowledge and reflect many of  the practices used by leading commercial companies, there is room for  improvement. Criteria used for getting manufacturing processes under  control are still not specific enough, allowing demonstration of controls to  occur too late in the process—after the milestone C decision authorizing  low-rate initial production—whereas commercial firms require that critical  processes be in control earlier. While MRLs represent positive change,  unless these criteria are strengthened at the time a production decision is  made, DOD will have missed an opportunity to reduce the risk of  continued cost growth on acquisition programs. Moreover, use of MRLs  would be enhanced by the development of analytical tools, such as those  used by Honeywell, to support MRL assessments.
A serious concern is that DOD’s in-house manufacturing workforce has  been diminishing for decades and that, therefore, could hamper successful  implementation of MRLs. Unless DOD develops long-range plans to build  its in-house manufacturing workforce, it may not be able to realize the full  potential of integrating manufacturing readiness levels into its processes.
section_title: Recommendations for Executive Action
paragraphs: To ensure that DOD is taking steps to strengthen and improve the  producibility and manufacturing readiness of technologies, weapon  systems, subsystems, or manufacturing processes, we recommend that the  Secretary of Defense do the following:    Require the assessment of manufacturing readiness across DOD  programs using consistent MRL criteria as basis for measuring,  assessing, reporting, and communicating manufacturing readiness and  risk on science and technology transition projects and acquisition  programs.
Direct the Office of the Director, Defense Research and Engineering to  examine strengthening the MRL criteria related to the process  capability and control of critical components and/or interfaces prior to  milestone C, or equivalent, for low-rate initial production decision.
Direct the Office of the Director, Defense Research and Engineering to  assess the need for analytical models and tools to support MRL  assessments.
Assess the adequacy of the manufacturing workforce knowledge and  skills base across the military services and defense agencies and  develop a plan to address current and future workforce gaps.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us written comments on a draft of this report. DOD  partially concurred with our recommendation to require the assessment of  manufacturing readiness across DOD programs using MRL criteria, and  concurred with our other recommendations. Their comments can be  found in appendix IV of this report.
In its comments, DOD partially concurred with the recommendation that  DOD programs be required to assess manufacturing readiness using  consistent MRL criteria as the basis for measuring, assessing, reporting,  and communicating manufacturing readiness and risk on science and  technology transition projects and acquisition programs. DOD cites the  Department of Defense Instruction 5000.02 as reflecting on manufacturing  throughout the acquisition life cycle and, specifically, establishing a  framework to continually assess and mitigate manufacturing risks. In its  remarks, DOD states that the manufacturing readiness criteria will be  tailored to programs and embedded into reviews and assessment  templates, including systems engineering reviews, preliminary design  reviews, and critical design reviews as well as acquisition phase exit  criteria.
While we are encouraged by DOD’s plans to incorporate manufacturing  readiness criteria into various assessments, we are concerned about the  absence of any reference to MRLs, which identify specific benchmarks for  each acquisition phase. It is unclear from DOD’s comments whether it  intends to use a common definition of manufacturing readiness as  acquisition phase exit criteria or whether the exit criteria will be decided  on a case-by-case basis. While tailoring to individual programs is  appropriate, tailoring must take place in the context of well-understood  criteria for moving from phase to phase. A hallmark of the commercial  programs we have looked at in this and other reviews is the reliance on  disciplined processes for assessing readiness to proceed into more costly  development and production phases. Firm criteria are needed to identify  and address producibility and manufacturing risks on a timely basis,  before they result in expensive production problems.
We also received technical comments from DOD, which have been  addressed in the report, as appropriate.
We are sending copies of this report to the Secretary of Defense; the  Secretary of the Army; Secretary of the Navy; Secretary of the Air Force;   Director, Missile Defense Agency; Director, Defense Contract Management  Agency; and Office of Management and Budget. In addition, the report will  be made available at no charge on the GAO Web site at  http://www.gao.gov.
If you, or your staff, have any questions concerning this report, please  contact me at (202) 512-4841. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. The major contributors are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: This report compares the Department of Defense (DOD) and its large  prime contractors’ manufacturing practices with those of leading  commercial companies—with a focus on improving the manufacturing of  defense weapon systems. Specifically, we assessed (1) the manufacturing  problems experienced by DOD, (2) how manufacturing readiness levels  (MRLs) can address DOD’s manufacturing problems, (3) how proposed  MRLs compare to manufacturing best practices of leading commercial  companies, and (4) the challenges and barriers to implementing MRLs at  DOD.
To identify the manufacturing problems experienced by DOD, we  performed an aggregate analysis of DOD programs from our annual  assessment database. We also conducted case studies of four programs  with known cost and schedule problems to make observations on the  types of problems DOD weapon systems may experience. The programs  we reviewed, along with the prime contractors responsible for developing  the systems, are    Joint Air-to-Surface Standoff Missile, an air-to-surface missile funded  by the Air Force and developed by Lockheed Martin;   Exoatmospheric Kill Vehicle, a ballistic-missile interceptor funded  by the Missile Defense Agency and developed by Raytheon;    Electromagnetic Aircraft Launch System, a launch system for  aircraft carriers funded by the Navy and developed by General  Atomics; and    H-1 helicopter upgrade, tactical utility and attack helicopters funded  by the Navy and developed by Bell Helicopter.
To evaluate the four DOD weapon programs, we examined program  documentation, such as acquisition decision memos and production  readiness reviews, and held discussions with manufacturing and systems  engineering officials from DOD program offices, the prime contractors,  and the Defense Contract Management Agency. Based on the information  gathered through interviews conducted and documentation synthesized,  we identified commonalities among the case studies.
To determine how MRLs can address the manufacturing problems  experienced by defense programs, we conducted interviews with officials  from the Office of Secretary of Defense, Office of the Director, Defense  Research & Engineering, Joint Defense Manufacturing Technology Panel  working group, National Center for Advanced Technologies, National  Defense Industrial Association, and Defense Acquisition University on  their observations on MRLs. We also reviewed the MRL deskbook, matrix  (risk areas), analyses, and training materials. We also conducted  interviews with Army, Navy, and Air Force officials who were involved or  familiar with the pilot tests of MRLs on various programs. The pilot  programs we examined at the military services include the following    Army—micro electro-mechanical systems inertial measurement unit,  micro electro-mechanical systems safety arm, ferroelectric and micro  electro-mechanical systems phase shifter, low-cost materials for  improved protection, rotorcraft cabin floor structure, embedded  sensors, and armor manufacturing;    Air Force—MQ-9 Reaper, F-35 Joint Strike Fighter induct inlet, high- durability hot exhaust structures, F-135 Pratt & Whitney propulsion  system, sensor hardening for tactical systems, and X-Band thin array  radar; and    Navy—P-8A aircraft.
To identify practices and criteria used by leading commercial companies  that can be used to improve DOD’s manufacturing process, we selected  and visited five companies based on several criteria: companies that (1)  make products that are comparable to DOD in terms of complexity, (2)  are recognized as leaders in developing manufacturing readiness criteria,  or (3) have won awards for their manufacturing best practices, or a mix of  the above. We met with these companies to discuss their product- development and manufacturing practices and the steps that they take to  mitigate manufacturing risks, ensure manufacturing readiness, and  improve supplier quality. We met with these companies to discuss their  product-development life cycle and the methods and metrics they use to  measure manufacturing maturity and producibility; manufacturing risk  management; supplier management; and the key factors in the company’s  successful manufacturing outcomes. We generalized much of the  information due to the proprietary nature of the data relating to their  manufacturing processes. Several companies provided data on specific  processes or products that they agreed to allow us to include in this  report. We reported on four of the five companies we visited. The five  companies we visited include the following    GE Aviation, a leading aerospace company, whose portfolio includes  commercial engines and services, military engines and services,  business and general aviation, engine components, and aviation  systems. We met with manufacturing and quality officials in Cincinnati,  Ohio, and discussed their manufacturing practices and manufacturing  maturity metrics. We also toured their Lean Lab production facility and  saw how these practices were applied.
GE Healthcare, which manufactures a range of products and services  that includes medical imaging and information technologies and  medical diagnostics. We met with manufacturing officials at their  Milwaukee, Wisconsin, plant and discussed their manufacturing  practices, including the development and manufacturing of their  Gemstone scintillator for use on advanced CT scanners.
Honeywell Aerospace, a global provider of integrated avionics,  engines, systems, and services for aircraft manufacturers, airlines,  business and general aviation, and military and space operations. We  met with manufacturing officials at their Phoenix, Arizona, facility and  discussed their manufacturing maturity processes and the models and  tools they used to assess this.
Siemens Mobility, a division of Siemens that develops and builds  light rail cars for the North American market. We met with  manufacturing and procurement officials at their Sacramento,  California, manufacturing and assembly plant to discuss the  manufacturing processes used in building their rail cars and their  supplier management practices.
Toyota Motor Engineering and Manufacturing is responsible for  Toyota’s engineering design, development and manufacturing activities  in North America. We met with officials in their production engineering  division in Erlanger, Kentucky, and also in their Toyota Technical  Center located in Ann Arbor, Michigan, and discussed their vehicle  development process and their methods for assuring supplier quality.
At each of the companies, we interviewed senior management officials  knowledgeable about the manufacturing methods, techniques, and  practices used throughout manufacturing and product development to  ensure manufacturing maturity and producibility of their products. In  particular, we discussed their (1) product development life cycle and the  methods, metrics, and tools used to determine manufacturing maturity  and producibility, (2) methods for identifying and mitigating risks in  manufacturing a product, and (3) methods for supplier management to  provide steady supply of quality parts.
In addition, we compared the practices of commercial firms to two major  defense weapon systems known to be producing systems within cost and  schedule goals and with successful manufacturing outcomes. To evaluate  these two programs, we examined program documentation and held  discussions with program and contracting officials. The two systems we  reviewed, along with the prime contractors responsible for developing the  systems, are    Lakota aircraft, a light utility helicopter that conducts noncombat  missions, funded by the Army and developed by the European  Aeronautic Defence and Space Company; and    Standard Missile 3 Block 1A, a ship-based antiballistic missile,  funded by the Missile Defense Agency and developed by Raytheon.
To determine the challenges and barriers to MRL implementation efforts,  we interviewed officials who were involved with the draft policy to  standardize MRLs, as well as the military-service policy organizations that  commented on the proposal. We also synthesized the information  gathered at the various levels throughout the defense community to  determine the issues surrounding MRLs as well as their merits. These  DOD organizations include the Office of the Secretary of Defense, each of  the military-service policy groups and program offices, the Office of the  Director, Defense Research and Engineering, Systems and Software  Engineering, and Joint Defense Manufacturing Technology Panel. To  obtain an understanding of the workforce challenges in manufacturing, we  reviewed selected documentation—such as Defense Science Board  studies—and interviewed officials at each of the military services and  Defense Contract Management Agency who were responsible for  workforce planning activities and revitalization initiatives.
We conducted this performance audit from January 2009 to February 2010  in accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Manufacturing Readiness Level (MRL) Definitions
paragraphs: MRL 1—Basic Manufacturing Implications Identified  This is the lowest level of manufacturing readiness. The focus is to address  manufacturing shortfalls and opportunities needed to achieve program  objectives. Basic research (i.e., budget activity 6.1 funds) begins in the  form of studies.
This level is characterized by describing the application of new  manufacturing concepts. Applied research (i.e., budget activity 6.2 funds)  translates basic research into solutions for broadly defined military needs.  Typically this level of readiness in the science and technology  environment includes identification, paper studies, and analysis of  material and process approaches. An understanding of manufacturing  feasibility and risk is emerging.
MRL 3—Manufacturing Proof of Concept Developed  This level begins the validation of the manufacturing concepts through  analytical or laboratory experiments. This level of readiness is typical of  technologies in the science and technology funding categories of Applied  Research and Advanced Development (i.e., budget activity 6.3 funds).  Materials or processes, or both, have been characterized for  manufacturability and availability but further evaluation and  demonstration is required. Experimental hardware models have been  developed in a laboratory environment that may possess limited  functionality.
MRL 4—Capability to Produce the Technology in a Laboratory  Environment  This level of readiness is typical for science and technology programs in  the budget activity 6.2 and 6.3 categories and acts as exit criteria for the  materiel solution analysis phase approaching a milestone A decision.  Technologies should have matured to at least technology readiness level 4.  This level indicates that the technologies are ready for the technology- development phase of acquisition. At this point, required investments,  such as manufacturing technology development, have been identified.  Processes to ensure manufacturability, producibility, and quality are in  place and are sufficient to produce technology demonstrators.  Manufacturing risks have been identified for prototype build, and  mitigation plans are in place. Target cost objectives have been established  and manufacturing cost drivers have been identified. Producibility  assessments of design concepts have been completed. Key design  performance parameters have been identified as well as any special  tooling, facilities, material handling, and skills required.
MRL 5—Capability to Produce Prototype Components in a  Production-Relevant Environment  This level of maturity is typical of the midpoint in the technology- development phase of acquisition, or in the case of key technologies, near  the midpoint of an advanced technology-demonstration project.  Technologies should have matured to at least technology readiness level 5.  The industrial base has been assessed to identify potential manufacturing  sources. A manufacturing strategy has been refined and integrated with  the risk-management plan. Identification of enabling/critical technologies  and components is complete. Prototype materials, tooling and test  equipment, as well as personnel skills, have been demonstrated on  components in a production-relevant environment, but many  manufacturing processes and procedures are still in development.  Manufacturing technology development efforts have been initiated or are  ongoing. Producibility assessments of key technologies and components  are ongoing. A cost model has been constructed to assess projected  manufacturing cost.
MRL 6—Capability to Produce a Prototype System or Subsystem in  a Production-Relevant Environment  This MRL is associated with readiness for a milestone B decision to initiate  an acquisition program by entering into the engineering and manufacturing  development phase of acquisition. Technologies should have matured to at  least technology readiness level 6. It is normally seen as the level of  manufacturing readiness that denotes completion of science and  technology development and acceptance into a preliminary system design.  An initial manufacturing approach has been developed. The majority of  manufacturing processes have been defined and characterized, but there  are still significant engineering or design changes, or both, in the system  itself. However, preliminary design of critical components has been  completed and producibility assessments of key technologies are  complete. Prototype materials, tooling and test equipment, as well as  personnel skills have been demonstrated on systems or subsystems, or  both, in a production-relevant environment. A cost analysis has been  performed to assess projected manufacturing cost versus target cost  objectives and the program has in place appropriate risk reduction to  achieve cost requirements or establish a new baseline. This analysis  should include design trades. Producibility considerations have shaped  system-development plans. Industrial capabilities assessment for  milestone B has been completed. Long-lead and key supply-chain elements  have been identified. All subcontractors have been identified.
MRL 7—Capability to Produce Systems, Subsystems, or  Components in a Production-Representative Environment  This level of manufacturing readiness is typical for the midpoint of the  engineering and manufacturing-development phase leading to the post- critical design review assessment. Technologies should be maturing to at  least technology readiness level 7. System detailed design activity is  underway. Material specifications have been approved and materials are  available to meet the planned pilot-line build schedule. Manufacturing  processes and procedures have been demonstrated in a production- representative environment. Detailed producibility trade studies and risk  assessments are underway. The cost model has been updated with  detailed designs, rolled up to system level, and tracked against allocated  targets. Unit-cost reduction efforts have been prioritized and are  underway. The supply chain and supplier quality assurance have been  assessed and long-lead procurement plans are in place. Production tooling  and test equipment design and development have been initiated.
MRL 8—Pilot-Line Capability Demonstrated; Ready to Begin Low- Rate Initial Production  This level is associated with readiness for a milestone C decision, and  entry into low-rate initial production. Technologies should have matured  to at least technology readiness level 7. Detailed system design is  essentially complete and sufficiently stable to enter low-rate production.  All materials are available to meet the planned low-rate production  schedule. Manufacturing and quality processes and procedures have been  proven in a pilot-line environment and are under control and ready for  low-rate production. Known producibility risks pose no significant  challenges for low-rate production. The engineering cost model is driven  by detailed design and has been validated with actual data. The Industrial  Capability Assessment for milestone C has been completed and shows that  the supply chain is established and stable.
MRL 9—Low-Rate Production Demonstrated; Capability in Place to  Begin Full-Rate Production  At this level, the system, component, or item has been previously  produced, is in production, or has successfully achieved low-rate initial  production. Technologies should have matured to at least technology  readiness level 9. This level of readiness is normally associated with  readiness for entry into full-rate production. All systems- engineering/design requirements should have been met such that there are  minimal system changes. Major system design features are stable and have  been proven in test and evaluation. Materials are available to meet planned  rate production schedules. Manufacturing process capability in a low-rate  production environment is at an appropriate quality level to meet design  key-characteristic tolerances. Production risk monitoring is ongoing. Low- rate initial production cost targets have been met, with learning curves  validated. The cost model has been developed for the full-rate production  environment and reflects the effect of continuous improvement.
MRL 10—Full-Rate Production Demonstrated and Lean Production  Practices in Place  This is the highest level of production readiness. Technologies should have  matured to at least technology readiness level 9. This level of  manufacturing is normally associated with the production or sustainment  phases of the acquisition life cycle. Engineering/design changes are few  and generally limited to quality and cost improvements. System,  components, or items are in full-rate production and meet all engineering,  performance, quality, and reliability requirements. Manufacturing process  capability is at the appropriate quality level. All materials, tooling,  inspection and test equipment, facilities, and manpower are in place and  have met full-rate production requirements. Rate production unit costs  meet goals, and funding is sufficient for production at required rates. Lean  practices are well established and continuous process improvements are  ongoing.
section_title: Appendix III: Manufacturing Readiness Level (MRL) Threads and Subthreads (Risk Areas)
paragraphs: Appendix  (MRL)
III: Manufacturing Readiness Level  Threads and Subthreads (Risk Areas)
Potential  sources  identiﬁ ed  for technol- ogy needs.  Understand  state of the  art.
Industrial base  capabilities  surveyed and  known; gaps/ risks identiﬁ ed  for preferred  concept, key  technologies,  components,  and/or key  processes.
Industrial base  assessment  initiated to  identify poten- tial manufac- turing sources.  Sole/ single/ foreign source  vendors have  been identiﬁ ed  and planning  has begun to  minimize risks.
Industrial Capa- bilities Assess- ment (ICA) for  Milestone (MS)  B has been com- pleted.  Indus- trial capability in  place to support  manufacturing  of development  articles.  Plans to  minimize sole/  foreign sources  complete.  Need  for sole/single/ foreign sources  justiﬁ ed.  Poten- tial alternative  sources identi- ﬁ ed.
Industrial  capabil- ity to support  production  has been ana- lyzed.  Sole/ single/ for- eign sources  stability is  assessed/ monitored.  Developing  potential  alternate  sources as  necessary.
ICA for MS  C has been  completed.  Industrial  capability  is in place  to support  Low Rate  Initial  Produc- tion (LRIP).  Sources are  available,  multi- sourcing  where cost- effective or  necessary  to mitigate  risk.
Industrial  capability  is in place  to support  start of Full  Rate Produc- tion (FRP).
Industrial  capability  supports  FRP and is  assessed  to support  modiﬁ ca- tions,  upgrades,  surge  and other  potential  manufactur- ing require- ments.
Manu- facturing  technology  concepts  identiﬁ ed  through ex- periments/ models.
Manufactur- ing Science  and Advanced  Manufacturing  Technology  requirements  identiﬁ ed.
Required  manufacturing  technology  development  efforts initiated  as applicable.
Manufacturing  technology  efforts continu- ing. Required  manufacturing  technology  development  solutions dem- onstrated in a  production-rele- vant environ- ment.
Manufactur- ing technol- ogy efforts  continuing.  Required  manu- facturing  technology  development  solutions  demon- strated in a  production- representative  environment.
Primary  manu- facturing  technol- ogy efforts  concluding  and some  improve- ment  efforts  continuing.  Required  manu- facturing  technology  solutions  validated  on a pilot  line.
Manufactur- ing technol- ogy process  improve- ment efforts  initiated for  FRP.
Manu- facturing  technology  continuous  process im- provements  ongoing.
Produc- ibility  improve- ments  imple- mented  on system.
subsections: section_title: Should be assssed
paragraphs: experimen- tal results. completed.  Results con- sidered in  selection of  preferred de- sign concepts  and reﬂ ected  in Technology  Development  Strategy (TDS)  key com- ponents/  technologies. initiated as ap- propriate.  On- going design  trades consider  manufactur- ing processes  and industrial  base capabil- ity constraints.  Manufactur- ing processes  assessed for  capability to  test and verify  in production,  and inﬂ uence  on operations  & support. technologies/ components  completed.  Results used to  shape Acquisi- tion Strategy  (AS), Systems  Engineering  Plan (SEP),  Manufacturing  and Produc- ibility plans, and  planning for  Engineering and  Manufacturing  Development   (EMD) or tech- nology insertion  programs.  Preliminary  design choices  assessed against  manufactur- ing processes  and industrial  base capabil- ity constraints.  Producibility  enhancement  efforts (e.g.,  Design for  Manufacturing  (DFX)) initiated. and related  manufactur- ing process  capability  completed.  Producibility  enhancement  efforts (e.g.,  DFX) ongoing  for optimized  integrated  system.  Manufactur- ing processes  reassessed as  needed for  capability to  test and verify  potential in- ﬂ uence on  operations &  support.
Known  producibil- ity issues  have been  resolved  and pose  no signiﬁ - cant risk for  LRIP. during LRIP.  Producibility  issues/ risks  discovered  in LRIP  have been  mitigated  and pose no  signiﬁ cant  risk for FRP. produc- ibility im- provements  ongoing.  All  modiﬁ ca- tions,  upgrades,  Diminishing  Manufactur- ing Sources  and Material  Shortages  (DMSMS),  and other  changes  assessed for  produc- ibility.
System al- located baseline  established.  System and  subsystem  preliminary  design sufﬁ cient  for EMD.  All  enabling/critical  technologies/ components  have been  demonstrated.  Preliminary de- sign KCs deﬁ ned.
Detailed  design of  product  features  and inter- faces is  complete.  All product  data  essential  for system  manu- facturing  has been  released.
section_title: Should be assssed
paragraphs: technical re- quirements  evaluated.
Performance  Parameter  (KPPs)  identiﬁ ed  for preferred  systems con- cept.  System  characteristics  and measures  to support  required capa- bilities identi- ﬁ ed.  Form, ﬁ t,  and function  constraints  identiﬁ ed, and  manufacturing  capabilities  identiﬁ ed for  preferred sys- tems concepts. design Critical  Characteristic  (KCs) initiated.  Product data  required for  prototype  component  manufacturing  released. component  manufactur- ing has been  released.  Potential KC  risk issues  have been  identiﬁ ed and  mitigation  plan is in  place.
Design  change  trafﬁ c does  not sig- niﬁ cantly  impact  LRIP.  KCs  are attain- able based  upon  pilot line  demonstra- tions.
Conﬁ gura- tion Audit  (PCA) or  equivalent  complete as  necessary.  Design  change traf- ﬁ c is limited.  All KCs are  controlled  in LRIP to  appropri- ate quality  levels. obsoles- cence.  All  KCs are  controlled  in FRP to  appropri- ate quality  levels.
Initial cost  targets  and risks  identiﬁ ed.  High level  process  chart model  developed.  Technology  cost models  developed  for new  process  steps and  materials  based on  experi- ments.
Manufacturing,  material and  special require- ment cost  drivers identi- ﬁ ed. Detailed  process chart  cost models  driven by pro- cess variables.  Cost driver  uncertainty  quantiﬁ ed.
Prototype  components  produced in  a produc- tion relevant  environment,  or simulations  drive end- to-end cost  models. Cost  model includes  materials,  labor, equip- ment, tooling/ (Special Test  Equipment  (STE), setup,  yield/ scrap/ rework, Work in  Process (WIP),  and capabil- ity/capacity  constraints).
Cost model  updated with  design require- ments, material  speciﬁ cations,  tolerances, inte- grated master  schedule, results  of system/ subsystem  simulations and  production rel- evant prototype  demonstrations.
Cost model  updated with  the results  of systems/ sub-systems  produced in  a production- representative  environment  and with pro- duction plant  layout and  design and  obsolescence  solutions.
Cost  models  updated  with results  of pilot line  build.
FRP cost  model up- dated with  result of  LRIP build.
Cost model  validated  against  actual FRP  cost.
section_title: Should be assssed
paragraphs: Sensitiv- ity analysis  conducted  to deﬁ ne  cost drivers  and produc- tion devel- opment  strategy (i.e.,  lab to pilot  to factory).
Producibility  cost risks as- sessed. Initial  cost models  support Analy- sis of Alterna- tives (AoA) and  Alternative  Systems  Review (ASR).
Costs  analyzed using  prototype  component ac- tuals to ensure  target costs  are achievable.  Decisions re- garding design  choices, make/ buy, capacity,  process capa- bility, sources,  quality, KCs,  yield/rate, and  variabil- ity inﬂ uenced  by cost  models.
Costs analyzed  using prototype  system/subsys- tem actuals to  ensure target  costs are achiev- able. Allocate  cost targets to  subsystems.  Cost reduction  and avoid- ance strategies  developed.
Manufactur- ing costs  rolled up to  system/sub- system level  and tracked  against tar- gets. Detailed  trade studies  and engineer- ing change  requests  supported  by cost esti- mates. Cost  reduction and  avoidance  strategies  underway.
Costs  analyzed  using pilot  line actuals  to ensure  target  costs are  achievable.  Manufac- turing cost  analysis  supports  proposed  changes to  require- ments or  conﬁ gura- tion.  Cost  reduction  initiatives  ongoing.
LRIP cost  goals met  and learn- ing curve  analyzed  with actual  data. Cost  reduction  initiatives  ongoing.  Touch labor  efﬁ ciency  analyzed  to meet  production  rates and  elements  of inef- ﬁ ciency are  identiﬁ ed  with plans  in place for  reduction.
FRP cost  goals  met. Cost  reduction  initiatives  ongoing.
Program/ projects  have  reasonable  budget  estimates  for reaching  MRL 4 by  MS A.
Manufacturing  technology  initiatives iden- tiﬁ ed to reduce  costs. Program  has reasonable  budget esti- mate for reach- ing MRL 6 by  MS B. Estimate  includes capi- tal investment  for production- relevant  equipment.  All outstand- ing MRL 4  risk areas  understood,  with approved  mitigation  plans in place.
Program has  updated bud- get estimate  for reaching  MRL 6 by MS B.  All outstand- ing MRL 5  risk areas  understood,  with approved  mitigation  plans in place.
Program has  reasonable  budget estimate  for reaching  MRL 8 by MS  C. Estimate  includes capital  investment for  production- representative  equipment by  CDR and pilot  line equipment  by MS C. All  outstanding  MRL 6 risk areas  understood,  with approved  mitigation plans  in place.
Program  has updated  budget  estimate for  reaching MRL  8 by MS C. All  outstanding  MRL 7 risk  areas under- stood, with  approved  mitigation  plans in place.
Program  has  reasonable  budget  estimate  for reach- ing MRL 9  by the FRP  decision  point.  Estimate  includes  investment  for LRIP  and FRP. All  outstand- ing MRL 8  risk areas  under- stood, with  approved  mitigation  plans in  place.
Program has  reasonable  budget  estimate  for FRP. All  outstanding  MRL 9 risk  areas under- stood, with  approved  mitigation  plans in  place.
Production  budgets  sufﬁ cient for  production  at required  rates and  schedule  to support  funded  program.
section_title: Should be assssed
paragraphs: Material  properties  validated  and as- sessed  for basic  manufactur- ability using  experi- ments.
Survey deter- mines that  the projected  material has  been produced  in a laboratory  environment. Survey deter- mines that  the projected  material has  been produced  in a laboratory  environment.
Materials  have been  manufactured  or produced  in a prototype  environment  (maybe in a  similar applica- tion/program).   Maturity  efforts in place  to address new  material pro- duction risks  for technology  demonstration.
Mate- rial maturity  veriﬁ ed through  technology  demonstration  articles. Prelimi- nary material  speciﬁ cations  in place and  material proper- ties have been  adequately  characterized.
Material  maturity suf- ﬁ cient for  pilot line  build. Material  speciﬁ cations  approved.Ma- terial maturity  sufﬁ cient for  pilot line  build. Material  speciﬁ cations  approved.
Materials  proven and  validated  during  EMD as  adequate  to sup- port LRIP.  Material  speciﬁ ca- tion stable.
Material is  proven and  controlled  to speciﬁ ca- tion in LRIP. Material is  proven and  controlled  to speciﬁ ca- tion in LRIP.
Material is  proven and  controlled  to speciﬁ ca- tion in FRP. Material is  proven and  controlled  to speciﬁ ca- tion in FRP.
Material  scale-up  issues  identiﬁ ed.
Projected lead  times have  been identiﬁ ed  for all difﬁ cult- to-obtain,  difﬁ cult-to- process, or  hazardous  materials.  Quantities  and lead times  estimated.
Availability is- sues addressed  for prototype  build. Signiﬁ - cant material  risks identiﬁ ed  for all materi- als. Planning  has begun to  address  scale-up issues.
Availability  issues addressed  to meet EMD  build. Long-lead  items identiﬁ ed.  Potential obso- lescence issues  identiﬁ ed.
Availabil- ity issues  addressed  to meet  EMD builds.  Long-lead  procurement  identiﬁ ed/  planned for  LRIP.  Obsoles- cence plan in  place.
Long-lead  procure- ment  initiated  for LRIP.  Availability  issues pose  no signiﬁ - cant risk for  LRIP.
Long-lead  procure- ment initi- ated for FRP.  Availability  issues pose  no signiﬁ - cant risk for  FRP.
Program  is in FRP,  with no  signiﬁ cant  material  availability  issues.
Initial as- sessment  of potential  supply  chain capa- bility.
Survey  completed  for potential  supply chain  sources.
Potential  supply chain  sources identi- ﬁ ed.
Supply chain  plans in place  (e.g. teaming  agreements and  so forth) leading  to an EMD con- tract award.
Effective  supply chain  management  process in  place. Assess- ment of criti- cal ﬁ rst tier  supply chain  completed.
Supply  chain ade-  quate to  support  LRIP.  As- sessment  of critical  second and  lower tier  supply chain  completed.
Supply  chain is  stable and  adequate to  support FRP.  Long-term  agreements  in place  where  practical.
Supply chain  proven and  supports  FRP require- ments.
Special  handling  procedures  applied in  pilot line  environ- ment.
Special  handling  procedures  effectively  implement- ed in FRP.
section_title: Should be assssed
paragraphs: applied  in the lab.  Special  handling  concerns  assessed. handling  requirements  identiﬁ ed. handling  requirement  gaps identiﬁ ed.  New special  handling pro- cesses demon- strated in lab  environment. requirement  gaps complete. ment.  Special  handling  procedures  developed  and anno- tated on work  instructions.
Special  handling  procedures  demon- strated in  EMD or  technology  insertion  programs.  Special han- dling issues  pose no  signiﬁ cant  risk for LRIP.  All work  instructions  contain  special  handling  provisions,  as required. handling  procedures  demonstrat- ed in LRIP.    Special han- dling issues  pose no  signiﬁ cant  risk for FRP.
Identiﬁ ca- tion of  proposed  manu- facturing  concepts or  produc- ibility needs  based on  high-level  process  ﬂ owchart  models.
Production  modeling and  simulation  approaches  for process or  product are  identiﬁ ed.
Initial simula- tion models  (product or  process) de- veloped at the  component  level.
Initial simulation  models devel- oped at the  subsystem or  system level.
Simulation  models used  to determine  system  constraints  and identify  improvement  opportunities.
Simulation  models veri- ﬁ ed by pilot  line build.  Results used  to improve  process and  determine  that LRIP  require- ments can  be met.
Simulation  model veri- ﬁ ed by LRIP  build, assists  in manage- ment of  LRIP and  determines  that FRP re- quirements  can be met.
Simula- tion model  veriﬁ ed by  FRP build.  Production  simulation  models used  as a tool  to assist in  manage- ment of FRP.
Document  high-level  manufac- turing  processes.  Critical  manufactur- ing process- es identiﬁ ed  through  experimen- tation.
Complete  a survey to  determine  the current  state of critical  processes.
Maturity has  been assessed  on similar  processes  in produc- tion. Process  capability  requirements  have been  identiﬁ ed for  pilot line, LRIP  and FRP.
Manufacturing  processes dem- onstrated in pro- duction-relevant  environment.  Begin collecting  or estimating  process capabil- ity data from  prototype build.
Manufactur- ing processes  demon- strated in a  production- representative  environment.  Continue  collecting or  estimating  process capa- bility data.
Manu- facturing  processes  veriﬁ ed for  LRIP on a  pilot line.  Process  capability  data from  pilot line  meets  target.
section_title: Should be assssed
paragraphs: to show FRP  impact and  potential for  continuous  improve- ment.
FRP objec- tives.
Initial  estimates of  yields and  rates based  on experi- ments or  state of the  art.
Yield and rates  assessment on  proposed/simi- lar processes  complete and  applied within  AoA.
Target yields  and rates  established for  pilot line, LRIP,  and FRP. Yield  and rate issues  identiﬁ ed.  Improvement  plans devel- oped/initiated.
Yields and  rates from  production-rele- vant environ- ment evaluated  against targets  and the results  feed improve- ment plan.
Yields and  rates from  production- representative  environment  evaluated  against pilot  line targets  and the  results feed  improvement  plans.
Pilot line  targets  achieved.  Yields  and rates  required to  begin LRIP  veriﬁ ed  using pilot  line articles.  Improve- ment plans  ongo- ing and  updated.
LRIP yield  and rate  targets  achieved.  Yield im- provements  ongoing.
FRP yield  and rate  targets  achieved.  Yield im- provements  ongoing.
Quality strate- gy identiﬁ ed as  part of the TDS  and included  in SEP.
Quality strat- egy updated  to reﬂ ect KC  identiﬁ cation  activities.
Initial quality  plan and quality  management  system is in  place. Quality  risks and metrics  have been  identiﬁ ed. -lauQ ity targets  established.  Demonstrate  ability to  collect and  analyze  quality data  (process and  system) in the  production- representative  environment.
Qual- ity targets  demon- strated on  pilot line.  Continuous  quality im- provement  ongoing.  Supplier  products  have com-  pleted  qualiﬁ - cation  testing and  ﬁ rst-article  inspection.  Supplier  products  pass accep- tance test- ing at a rate  adequate to  begin LRIP.
Quality tar- gets veriﬁ ed  on LRIP line.  Continu- ous quality  improvement  ongoing.  Supplier  products  pass accep- tance testing  at a rate  adequate to  transition to  FRP.
Quality tar- gets veriﬁ ed  on FRP line.  Continuous  quality im- provement  ongoing.
section_title: Should be assssed
paragraphs: New  manufactur- ing skills  identiﬁ ed.
Manufactur- ing skill sets  identiﬁ ed and  production  workforce  requirements  (technical and  operational)  evaluated as  part of AoA.  Determine  availabil- ity of process  development  workforce for  the TDP.
Skill sets  identiﬁ ed and  plans devel- oped to meet  prototype  and produc- tion needs.  Special skills  certiﬁ cation  and training  requirements  established.
Manufacturing  workforce skills  available for  production in  a relevant envi- ronment. Iden- tify resources  (quantities and  skill sets) and  develop initial  plans to achieve  requirements  for pilot line and  production.
Manufactur- ing workforce  resource  requirements  identiﬁ ed  for pilot  line. Plans  developed to  achieve pilot  line require- ments. Plans  updated to  achieve LRIP  workforce  requirements.  Pilot line  workforce  trained on  representative  environment.
Manu- facturing  workforce  resource re- quirements  identiﬁ ed  for LRIP.  Plans  developed  to achieve  LRIP re- quirements.  Plans  updated to  achieve FRP  workforce  require- ments. LRIP  personnel  trained on  pilot line  where pos- sible.
LRIP  personnel  requirements  met. Imple- ment plan to  achieve FRP  workforce re- quirements.
FRP person- nel require- ments met.  Production  workforce  skill sets  maintained  due to  attrition of  workforce.
Tooling/STE/ SIE require- ments are  considered as  part of AoA.
Identify tool- ing and Special  Test Equip- ment / Special  Inspection  Equipment  (STE/SIE)  requirements  and provide  supporting  rationale and  schedule.
Prototype tool- ing and STE/SIE  concepts dem- onstrated in pro- duction relevant  environment.  Production tool- ing and STE/SIE  requirements  developed. noitcudorP tooling and  STE/SIE  design and  development  efforts under- way. Manu- facturing  equipment  maintenance  strategy  developed.
All tooling,  test, and  inspection  equipment  proven on  pilot line  and re- quirements  identiﬁ ed  for LRIP.  Manu- facturing  equipment  mainte- nance  demon- strated on  pilot line.
All tooling,  test, and  inspection  equipment  proven in  LRIP and  requirements  identiﬁ ed for  FRP. Manu- facturing  equipment  maintenance  schedule  demon- strated.
Proven tool- ing, test, and  inspection  equipment  in place to  support  maximum  FRP. Planned  equipment  mainte- nance  schedule  achieved.
Specialized  facility re- quirements/ needs  identiﬁ ed.
Manufacturing  facilities identi- ﬁ ed and plans  developed  to produce  prototypes.
Manufacturing  facilities identi- ﬁ ed and plans  developed to  produce pilot  line build.
section_title: Should be assssed
paragraphs: evaluated as  part of AoA. produce LRIP  build. facilities  adequate  to begin  LRIP. Plans  in place to  support  transition  to FRP.
LRIP.  Capac- ity plans  adequate to  support FRP. maximum  FRP require- ments.
Manufactur- ing strategy  developed and  integrated  with acquisi- tion strat- egy.  Prototype  schedule risk  mitigation  efforts incor- porated into  TDS.
Manufactur- ing strategy  reﬁ ned based  upon preferred  concept. Proto- type schedule  risk mitigation  efforts initi- ated.
Initial manufac- turing approach  developed.  All system- design-related  manufacturing  events included  in Integrated  Master Plan/ Integrated  Master Schedule  (IMP/IMS).  Manufacturing  risk mitigation  approach for  pilot line or  technology  insertion pro- grams deﬁ ned. -unam laitinI facturing plan  developed.   Manufactur- ing planning  required to  achieve MRL  8 has been  included in  the IMP/IMS.  Manufactur- ing risks  integrated  into risk miti- gation plans.  Develop  initial work  instructions.  Effective  production  control sys- tem in place  to support  pilot line.
Manufac- turing plan  updated for  LRIP.  All key  manufac- turing risks  are identi- ﬁ ed and as- sessed with  approved  mitigation  plans in  place.  Work  instructions  ﬁ nalized.  Effective  production  control  system in  place to  support  LRIP.
Manufactur- ing plan  updated  for FRP.  All  manufactur- ing risks  tracked and  mitigated.  Effective  production  control sys- tem in place  to support  FRP.
All manufac- turing risks  mitigated.
Technology  development  article  component  list developed  with associ- ated lead-time  estimates.
Technology  development  part list matur- ing.  Make/ buy evalua- tions begin  and include  production  considerations  reﬂ ecting pilot  line, LRIP, and  FRP needs.  Lead times  and other risks  identiﬁ ed.
Most material  decisions com- plete (make/ buy), material  risks identiﬁ ed,  and mitigation  plans devel- oped.  Bill of  Materials (BOM)  initiated. yub/ekaM decisions  and BOM  complete  for pilot line  build.  Mate- rial planning  systems in  place for pilot  line build.
Make/buy  decisions  and BOM  complete  to sup- port LRIP.  Material  planning  systems in  place for  LRIP build.
Make/buy  decisions  and BOM  complete  to support  FRP.  Material  planning sys- tems in place  for FRP.
Mate- rial planning  systems  validated on  FRP build.
section_title: Appendix IV: Comments from the Department of Defense
paragraphs: DoD Response to GAO-10-439 Recommendations  GAO DRAFT REPORT DATED MARCH 12, 2010  GAO-10-439 (GAO CODE 120793)  “BEST PRACTICE: DOD CAN ACHIEVE BETTER OUTCOMES BY  STANDARDIZING THE WAY MANUFACTURING RISKS ARE MANAGED”
DEPARTMENT OF DEFENSE COMMENTS TO THE GAO  RECOMMENDATION 1:  The GAO recommends that the Secretary of Defense require  the assessment of manufacturing readiness across DoD programs using consistent MRL  criteria as a basis for measuring, assessing, reporting, and communicating manufacturing  readiness and risk on science and technology transition projects and acquisition  programs.
DOD RESPONSE:  Partially concur:  The Department of Defense recognizes that mature  manufacturing processes and readiness are critical to achieving predictable and successful  program outcomes.  It also recognizes the value in assessing manufacturing risks during  science and technology research on technologies planned to be incorporated into  acquisition programs.  Department of Defense Instruction (DoDI) 5000.02, Operation of  the Defense Acquisition System, dated 8 December 2008 reflects an increased focus on  manufacturing throughout the acquisition lifecycle for programs of all acquisition  categories.  Specifically, it establishes a framework to continually assess and mitigate  manufacturing risks during the Analysis of Alternatives, 2366b certifications to Congress,  Preliminary and Critical Design Reviews; and acquisition milestones.
The Department’s new manufacturing readiness criteria will form the basis for assessing  pertinent science and technology efforts, and acquisition programs throughout the  acquisition lifecycle on programs of all acquisition categories.   These criteria will be a  tool to identify relevant manufacturing risks which require mitigation.  These  manufacturing readiness criteria are expected to be tailored for programs and will be  included in the Department’s criteria for systems engineering technical reviews; the  Department’s templates for Preliminary Design Review/Critical Design Review reports;  and acquisition phase exit criteria.  These manufacturing readiness criteria will also be  assessed as part of the Program Support Reviews which the Department conducts on  Major Defense Acquisition Programs.  These reviews evaluate manufacturing as part of  an overall integrated program assessment.  These manufacturing readiness criteria and  products will be made available to government and industry.  Their use by the Services  on lower ACAT programs will also be encouraged.  The Navy’s Gate Review process  currently assesses manufacturing risks but is being updated with the new manufacturing  readiness criteria.
DoD Response to GAO-10-439 Recommendations  RECOMMENDATION 2:  The GAO recommends that the Secretary of Defense direct  the Office of the Director, Defense Research and Engineering to examine strengthening  the MRL criteria related to the process capability and control of critical components  and/or interfaces prior to the Milestone C low rate initial production decision.
DOD RESPONSE:  Concur.  Department of Defense Instruction 5000.02 directs that  programs at Milestone C have no significant manufacturing risks;  that manufacturing  processes have been effectively demonstrated in a pilot line environment; and  manufacturing processes are under control (if Milestone C is full-rate production).  While  the Department notes that all manufacturing processes do not warrant the same level of  process capability and control, appropriate levels of control are certainly warranted on a  case by case basis.
The Department will examine strengthening the manufacturing readiness criteria related  to process capability and control of critical components and/or interfaces prior to the  Milestone C low rate initial production decision.  However, program offices and  contractors should continue to have the latitude to jointly agree on the targets and specific  process control demonstrations  required on the pilot production line during the  Engineering and Manufacturing Development to ensure success.
RECOMMENDATION 3:  The GAO recommends that the Secretary of Defense direct  the Office of the Director, Defense Research and Engineering to assess the need for  analytical models and tools to support MRL assessments.
DOD RESPONSE:  Concur.  The Department will collaborate with government services,  contractors, and academia to capture knowledge and provide improved tools for  government and contractor usage in conducting assessments of manufacturing readiness  as part of systems engineering technical reviews and milestone reviews.
RECOMMENDATION 4:  The GAO recommends that the Secretary of Defense assess  the adequacy of the manufacturing workforce knowledge and skills base across the  military services and defense agencies and develop a plan to address current and future  workforce gaps   DOD RESPONSE:  Concur.  We agree that the production, quality and manufacturing  (PQM) career field has suffered erosion, as have other DoD career fields.  The USD  (AT&L) Director of Human Capital has launched a review of the PQM career field  design to identify the skills, knowledge and training required at each level of career  progression in order to develop training courses and evaluate progression of anticipated  DoD planned new hires.  The Department has started to implement hiring and retention  strategies to mitigate the potential loss in experienced, senior-level PQM talent and  increase the size of the manufacturing workforce.  As part of the Secretary’s growth  strategy and other initiatives, the PQM career field is projected to grow approximately  DoD Response to GAO-10-439 Recommendations  1,300 (13%) by FY2015.  Each of the military services and other DOD components has  been actively planning and deploying initiatives that support the DOD acquisition  workforce growth strategy.  Components have submitted planning inputs to OSD and to  the Defense Acquisition Workforce Senior Steering Board, and growth is underway.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Key contributors to this report were Karen Zuckerstein, Assistant Director;  John M. Ortiz, Jr.; Beverly Breen; Leigh Ann Nally; Dr. W. Kendal Roberts;  Andrea Bivens; Kristine Hassinger; Kenneth Patton; Bob Swierczek; and  Dr. Timothy Persons, Chief Scientist.
section_title: Related GAO Products
paragraphs: Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-10-388SP. Washington, D.C.: March 30, 2010.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-09-326SP. Washington, D.C.: March 30, 2009.
Best Practices: Increased Focus on Requirements and Oversight Needed  to Improve DOD’s Acquisition Environment and Weapon System  Quality. GAO-08-294. Washington, D.C.: February 1, 2008.
Best Practices: Stronger Practices Needed to Improve DOD Technology  Transition Processes. GAO-06-883. Washington, D.C., September 14, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,  2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon  System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,  1996.
Why Some Weapon Systems Encounter Production Problems While  Others Do Not: Six Case Studies. GAO/NSIAD-85-34. Washington, D.C.:  May 24, 1985.