Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed
Released: 2010-04-22T00:00:00

Letter

Background

In recognition of the lack of manufacturing knowledge at key decision points and the need to develop more affordable weapon systems, DOD made recent changes to its policy. In 2008, the department made constructive changes to its policy instruction on operation of the defense acquisition system. It also developed MRLs as a measure that could strengthen the way the department manages and develops manufacturing- intensive systems. In 2004, the Joint Defense Manufacturing Technology Panel sponsored a joint defense and industry working group to design and develop MRLs for programs across DOD. In May 2005, MRLs were first introduced to the defense community in DOD’s Technology Readiness Assessment Deskbook for science and technology and acquisition managers to consider.

As new manufacturing readiness criteria, MRLs are a measurement scale designed to provide a common metric and vocabulary for assessing manufacturing maturity and risk. MRL assessments identify the risks and manufacturing readiness of a particular technology, manufacturing process, weapon system, subsystem, or element of a legacy program at key milestones throughout the acquisition life cycle. There are 10 basic MRLs designed to be roughly congruent with comparable levels of technology readiness levels for ease of use and understanding. Table 1 shows the MRLs and basic definitions (see appendix II for the detailed MRL definitions).

The working group also developed a set of elements called “threads” to provide acquisition managers and those conducting assessments an understanding of the manufacturing risk areas (see table 2). For these threads, desired progress is defined for each MRL, to provide an understanding of risks as readiness levels increase from one MRL to the next. Conceptually, these threads are manufacturing elements that are essential to programs as they plan, prepare for, and manage the activities necessary to develop a product. For example, the materials thread requires an assessment of potential supplier capability by MRL 3 and an assessment of critical first-tier suppliers by MRL 7. Likewise, the manufacturing personnel thread calls for identifying new manufacturing skills by MRL 3 and identifying manufacturing workforce requirements for the pilot line by MRL 7.

As shown, each basic thread (risk area) has a description and general requirements for assessing risks for each thread. The working group further decomposed these MRL threads into subthreads to provide users a detailed understanding of the various kinds of manufacturing risks. See appendix III for a detailed breakdown of these threads (risk areas) for each MRL.

DOD’s Long-standing History of Manufacturing Problems

GAO has conducted an extensive body of work that highlights many of the manufacturing-related problems that both DOD and its prime contractors have faced. In many respects, DOD has recognized the nature of these problems throughout the years and has taken a number of proactive steps to address them. GAO’s work has drawn on lessons learned and best practices to recommend ways for DOD to improve the way it develops and manufactures its weapon systems. Examples from our reports include the following: In 1996, GAO reported the practices that world-class commercial organizations had adopted to more efficiently produce quality products, to improve DOD’s quality assurance program. DOD was spending $1.5 billion extra per year on military-unique quality assurance requirements for major acquisitions and billions more on cost and schedule overruns to correct problems. GAO concluded that repeated unstable designs, poor process controls, and poor transition to production caused the manufacturing quality problems. While DOD had taken some actions, its culture was cited as the biggest reason for slow adoption and unimplemented recommendations.

In 1998, GAO reported on best commercial practices to offer ways to improve the process DOD uses to manage suppliers engaged in developing and producing major weapon systems. In assessing defense contractors and two case studies of munitions programs, the report concluded that suppliers were critical in the amount of technological innovation they contribute to the final product.

In 2002, GAO reported on how best practices could offer improvements to the way DOD develops new weapon systems, primarily the design and manufacturing aspects of the acquisition process. DOD’s record showed a history of taking longer and spending more than planned to develop and acquire weapon system which reduced its buying power. The report identified and recommended best practices for capturing and using design and manufacturing knowledge early and new development processes that s, included high-level decision points and knowledge-based exit criter before key decisions on production are made. Essentially, one of the high-level decision points has become what GAO commonly refers to as Knowledge Point 3—the point when a program has demonstrated the manufacturing processes are mature. The report also recommended a best practice that includes a standard called the Process Capability Index (Cpk), a process performance measurement that quantifies how closely a product is running to its specification limits. The index indicates how well the processes statistical performance meets its control limit requirement.

In 2008, GAO reported on how DOD and its defense contractors can improve the quality of major weapon systems. We reported that if DOD continued to employ the same acquisition practices as it has in the past, the cost of designing and developing its systems could continue to exceed estimates by billions of dollars. Quality problems were identified as the cause for cost overruns, schedule delays, and reduced weapon-system availability. Like DOD prime contractors, leading commercial firms rely on many practices related to systems engineering, manufacturing, and supplier quality, but they were more disciplined, and had institutionalized processes to ensure quality.

Since 2003, GAO has issued a series of annual assessment reports on selected weapons programs, increasing from 77 to 96 programs reviewed. At $296 billion, the cumulative cost growth for DOD programs reported in 2009 was found to be higher than it had been five years earlier, and the overall performance of weapon system programs was still poor. Although the cost growth and the 22-month average delay in delivering initial capabilities was not attributed to manufacturing alone, the lack of production maturity was cited as one of three key knowledge areas contributing to the department’s cost growth, schedule delay, and performance problems.

Revised Policy Incorporates Manufacturing Best Practices

DOD’s December 2008 revision to its policy instruction on operation of the defense acquisition system incorporates a number of the best practices we identified in our previous work. The instruction covers the entire life cycle and considers manufacturing risks earlier in the acquisition life-cycle framework. In a November 2003 report on DOD’s May 2003 revision to its policy, we reported that much of the revised policy agrees with GAO’s extensive body of work and that of successful commercial firms. While we assessed DOD’s revised policy as providing a good framework for capturing knowledge about critical technologies, product design and manufacturing processes, we reported in 2006 that acquisition officials were not effectively implementing the acquisition policy’s knowledge- based process. We reported that the effective implementation of policy was limited by the absence of effective controls that require compliance and specific criteria for clearly demonstrating that acceptable levels of knowledge about technology, design, and manufacturing have been attained at critical junctures before making further investments in a program. We concluded that without specific criteria—or standards against which a judgment or decision is quantifiably based—decision makers are permitted to make decisions on the basis of subjective judgment. The December 2008 revised policy instruction establishes target maturity criteria for measuring risks associated with manufacturing processes at milestone decision points.

During the material solutions phase, prior to milestone A, the 2008 policy instruction requires the analysis of alternatives to assess “manufacturing feasibility.” During the technology development phase, prior to milestone B, the instruction states the following: Prototype systems or appropriate component-level prototyping shall be employed to “evaluate manufacturing processes.”

A successful preliminary design review will “identify remaining design, integration, and manufacturing risks.”

A program may exit the technology development phase when “the technology and manufacturing processes for that program or increment have been assessed and demonstrated in a relevant environment” and “manufacturing risks have been identified.”

After milestone B, one of the purposes of the engineering and manufacturing development phase is to “develop an affordable and executable manufacturing process.” The instruction says that: “the maturity of critical manufacturing processes” is to be described in a post- critical design review assessment; system capability and manufacturing process demonstration shall show “that system production can be supported by demonstrated manufacturing processes;” and the system capability and manufacturing process demonstration effort shall end, among other things, when “manufacturing processes have been effectively demonstrated in a pilot line environment, prior to milestone C.”

Finally, at milestone C, the instruction establishes two entrance criteria for the production and deployment phase, which include “no significant manufacturing risks” and “manufacturing processes under control (if Milestone C is full-rate production).” Low-rate initial production follows in order to ensure an “adequate and efficient manufacturing capability.” In order to receive full-rate production approval, the following must be shown: 1. “demonstrated control of the manufacturing process,” 2. “the collection of statistical process control data,” and 3. “demonstrated control and capability of other critical processes.”

Even with the updated policy instruction in place that includes guidance for most knowledge-based practices, inconsistent implementation has hindered DOD’s past efforts to reform its acquisition practices. For example, we reported in 2006 that DOD was not effectively implementing the knowledge-based approach process and evolutionary approach emphasized in its policy. While the policy outlined specific knowledge- based process of concept refinement and technology development to help ensure a sound business case is developed before committing to a new development program, we found that almost 80 percent of the programs we reviewed were permitted to bypass this process.

Manufacturing Problems Are Attributed to Several Factors during the Planning and Design Phases of Selected DOD Weapons Programs

Defense acquisition programs continue to have problems manufacturing weapon systems. As a result, systems cost far more and take far longer to produce than estimated. Many programs authorized to enter production experienced billions of dollars in cost growth after the authorization— nearly two-thirds of those programs reported increases in average procurement unit costs. Several factors contribute to these issues during the planning and design phases. These include the inattention to manufacturing during planning and design, poor supplier management, and lack of a knowledgeable manufacturing workforce. Essentially, some of these programs moved into production without considering manufacturing risks earlier in development. This hindered managers from later managing those risks until they became problematic, and also led to subsequent problems with supplier management, such as prime contractors conducting little oversight of suppliers. Some programs also had an inadequate workforce—in terms of insufficient knowledge and numbers—to effectively manage and oversee defense manufacturing efforts.

Manufacturing Contributed to Growth in Cost and Delays in Schedule

Defense acquisition programs continue to be troubled by unstable requirements, immature technology, and a lack of manufacturing knowledge early in design, resulting in more costly products that take longer to produce. Our 2009 annual assessment shows that total research and development costs were 42 percent higher than originally estimated. These higher costs reflect in part the learning that takes place as manufacturing processes are established and used to produce the first prototypes.

Even programs that have been authorized to begin production have experienced substantial cost growth after the production decision. Production performance can be measured by examining the cost growth as expressed in changes to average procurement unit cost. This represents the value DOD gets for the procurement dollars invested in a certain program and shows the net effect of procurement cost growth and quantity changes. Figure 1 shows the levels of average procurement unit- cost growth for selected major defense acquisition programs.

As indicated in figure 1, nearly two-thirds of programs that entered production after 2000 reported more than a 5 percent increase in average unit cost growth, while 32 percent of programs reported average unit cost growth that ranged from 11 percent to more than 15 percent. One program reported a 25 percent increase in average procurement unit cost. Further, 42 percent of those programs experienced production cost increases when procured quantities decreased or remained the same. For example, the Black Hawk helicopter’s 2007 production estimate had no increase in quantities since 2005, yet its production cost increased $2.3 billion, and average procurement unit cost rose by 13 percent. The Joint Air-to-Surface Standoff Missile had an 8 percent quantity decrease since the 2004 production decision; but the production costs increased by $561 million and average procurement unit cost increased by 25 percent.

As for schedule growth, DOD has continued to experience delays in delivering new or modified weapon systems to the warfighter. Over 50 percent of current programs in production have encountered some form of delay after the production decision, when manufacturing processes should be in control. Consequently, warfighters often must operate costly legacy systems longer than expected, find alternatives to fill capability gaps, or go without the capability altogether.

The four DOD weapon systems we selected for in-depth review with known cost, schedule, and performance problems reported several key factors that contributed to manufacturing problems. These include the inattention to manufacturing during planning and design, poor planning for supplier management, and lack of a knowledgeable manufacturing workforce. Capturing critical manufacturing knowledge during the planning and design phases before entering production helps to ensure that a weapon system will work as intended and can be manufactured efficiently to meet cost, schedule, and quality targets. The programs in our review often lacked manufacturing knowledge at key decision points, which led to cost growth and schedule delays. For example, the Joint Air- to-Surface Standoff Missile program—an autonomous, air-to-ground missile designed to destroy high-value targets—experienced a critical unit- cost breach due to missile reliability problems not being addressed early in the design phase. Also, the Electromagnetic Aircraft Launch System—a new catapult technology being developed for the Navy’s newest class of aircraft carriers—had experienced problems manufacturing compatible materials, which resulted in cost growth and schedule delays and was the focus of recent congressional interest. Figure 2 summarizes contributing factors for manufacturing problems experienced by the four DOD weapon systems.

As indicated, most of the programs had more than one major problem related to manufacturing. These issues illustrate the major problems we discussed with defense and contractor officials, but do not encompass all the manufacturing problems experienced by the programs. For example, recent Air Force study reports that manufacturing and quality assur ance requirements are not included in the contracts to develop weapon systems, which could affect the contractor’s approach to manufactu Officials from the Defense Contract Management Agency—a DOD component that works directly with defense suppliers to ensure that supplies and services are delivered on time, at projected cost, and meet performance requirements—also reported similar contra could affect contractor performance on manufacturing. ring.

Manufacturing Was Overlooked du Development

Each of the four programs we examined did not give manufacturing strong consideration during the early planning and design phases. Programs we re moved into production largely without considering manufacturing risks earlier in the acquisition process, as demonstrated by the experiences of the Exoatmospheric Kill Vehicle and the H-1 helicopter upgrade program The Exoatmospheric Kill Vehicle was designed to intercept and destroy high-speed ballistic missile warheads in mid-flight, while the H-1 upgrad program converts the attack helicopter and the utility helicopter to the AH-1Z and UH-1Y configurations, respectively. .

The Exoatmospheric Kill Vehicle program was put on an accelerated development schedule in response to a directive to develop and deploy, the earliest possible date, ballistic missile defense drawing on the best technologies available. According to the contractor, it bypassed some of its normal development-review processes to accelerate delivery of the vehicle, which also resulted in a high acceptance of manufacturing risks without sufficient identification and management of risk-mitigation p For example, the program went into production without completing qualification testing. In addition, the contractor continued to incorpo design changes while supplier production was ongoing, resulting in rework and disruption to the production line. Early lots of kill vehicles were built manually by engineers in the absence of automated productio processes, which caused dissimilarities among vehicles in the fleet and will make refurbishments difficult. lans.

For several reasons, the H-1 helicopter upgrade program did not inclu manufacturing in the early phases of planning and also proceeded to de production before its design was mature, according to the contracto First, the program underestimated the complexity of updating and remanufacturing the aircraft without historical drawings. The emphasis was placed on minimizing development costs and resources were not available to assess manufacturing challenges early in the redesign process. Furthermore, the program started low-rate production before completing operational evaluation testing. As a result, the problems uncovered during testing had to be corrected on aircraft that were on the assembly line. Also, constant change orders and factory bottlenecks, among other problems, affected program costs and schedules. The schedule pressure allowed little opportunity to remedy the manufacturing problems, resulting in more complicated and expensive fixes. Ultimately the schedule slowed and the costs increased to the point that the program abandoned the remanufacturing upgrade and, instead, opted to purchase newly manufactured aircraft cabins for the UH-1Y configuration. r.

Poor Planning Led t Supplier Problems

Inattention to manufacturing during planning and design led to subsequ problems with supplier management in two major defense acquisition programs we reviewed. Specifically, the prime contractors did not give adequate attention to managing their suppliers. For example, program officials for the Joint Air-to-Surface Standoff Missile told us that the responsibility for manufacturing processes and discipline shifted in the 1990s from the government to the defense contractors. The government started to rely on the prime contractor to ensure quality and reliability, particularly with subtier suppliers. In this case, the program office told that the prime contractor for the missile program relied on the subtier suppliers to self-report their capabilities and did not engage in effective oversight of their work, which led to defective parts. The program office recently recruited experts in manufacturing to help the prime contractor address their supplier problems more effectively.

Lack of Manufacturing Knowledge Contributed to Problems

Some DOD programs and prime contractors had an inadequate defense manufacturing workforce—both in terms of numbers and experience—to effectively manage and oversee manufacturing efforts, which resulted in schedule delays or cost inefficiencies. The manufacturing workforce includes occupations such as specialists in quality assurance, business, manufacturing engineering, industrial engineering, and production control. In many cases, the programs lacked manufacturing expertise early in development, which hindered the program’s ability to later manage manufacturing risks. For example, the contractor for the Electromagnetic Air Launch System did not have sufficient systems-engineering personnel involved in the design to help it transition from development to production. As a result, the program encountered schedule delays and cost increases. DOD conducted a program assessment review, which led the program office and contractor to increase systems engineering staff.

For the Exoatmospheric Kill Vehicle program, the contractor’s workforce and manufacturing processes could not readily undertake the rigors of production for a space-based capability, part of which must be manufactured in a clean room environment, and all of which commands rigorous processes and procedures due to highly technical designs. The contractor’s hourly assembly personnel were trained to build tactical missiles on a high-rate production line and were not sufficiently trained in the quality-control standards required by clean-room manufacturing, such as carefully controlling foreign-object debris, specially maintaining the clean room, and using a partner in certain high-level tasks to ensure all steps are properly followed. These standards were not institutionalized, and the contractor eventually had to modify its facilities and production standards to correct the manufacturing problems. The facility had to be retooled and reconfigured late in development. The contractor also experienced high turnover in its workforce due to the increasing demands associated with working in a clean-room environment and working long hours.

MRLs Have Been Proposed to Improve the Way DOD Identifies and Manages Manufacturing Risk and Readiness

The Joint Defense Manufacturing Technology Panel working group has proposed MRLs as new manufacturing readiness criteria that could improve weapon system outcomes by standardizing the way programs identify and manage manufacturing risks associated with developing and fielding advanced weapon systems. MRLs were first introduced to the defense community in DOD’s 2005 Technology Readiness Assessment Deskbook as an important activity for science and technology and acquisition managers to consider. An analysis by the working group shows that MRLs address many of the manufacturing issues not covered by DOD’s technical reviews, particularly reviews conducted in the early phases of acquisition. In their development, comprehensive efforts were undertaken to design and develop MRLs from DOD as well as industry resources. For example, the working group formulated MRLs from a manufacturing knowledge base of defense, industry, and academia to address two key areas of risk—immature product technologies and immature manufacturing capability. The working group also designed MRLs as a structured and disciplined approach for the way manufacturing risk and readiness is expected to be identified and assessed. The working group also developed a set of tools that include a deskbook, checklist, and a website to help managers and users apply MRLs and conduct assessments. In addition, the Army and Air Force report that their use of MRLs on pilot programs contributed to substantial cost benefits on a variety of programs, including major acquisition programs.

MRLs Were Developed from Knowledge-Based Resources on Manufacturing

To develop MRLs, the working group conducted comprehensive sessions with industry participants to ensure the metrics and vocabulary for assessing manufacturing readiness would be an all-inclusive body of knowledge. Officials stated that a mature set of manufacturing knowledge resources already existed but it was scattered and not consistently applied in a disciplined way that aligned with the DOD acquisition life-cycle framework. In their formulation, MRLs were developed from an extensive body of manufacturing knowledge that included, but was not limited to, the following defense, industry, and academic sources: DOD Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008), Navy best-practices manual for using templates on design and Air Force manufacturing development guide, military standards and specifications, and Malcolm Baldrige quality award criteria.

Other standards and technical sources were obtained from the Institute of Electrical and Electronics Engineers, the International Standards Organization on quality management systems, automotive industry quality standards, and the supplier model from the Massachusetts Institute of Technology.

Analysis Shows MRLs Address Manufacturing Gaps in DOD’s Technical Reviews

An analysis conducted by the working group shows that MRLs address many of the manufacturing gaps identified in several of DOD’s technical reviews that provide program oversight and determine how well programs are meeting expected goals, particularly the reviews conducted in the early acquisition phases. According to the working group, addressing these manufacturing gaps is fundamental to improving the way programs plan, design, and prepare for manufacturing. For example, the working group’s analysis shows that DOD’s current systems-engineering technical review checklist used for preliminary design reviews has only 27 of 759 total questions that deal with core manufacturing-related questions, whereas the MRL 6 assessment checklist for this juncture has 169 core manufacturing questions. More importantly, the technical review checklist did not address key manufacturing discipline in the areas of program management, systems engineering, requirements management, risk management, and program schedule. Similarly, the technical review checklist used for critical design reviews has only 22 of 824 total questions that deal with core manufacturing questions, whereas the MRL 7 assessment checklist for this juncture has 162 core questions. Core manufacturing disciplines were not addressed in the specific areas of management metrics, manufacturing planning, requirements management, system verification, and other areas. Finally, DOD’s technical review checklist used for production readiness reviews has 194 of 613 total questions that deal with core manufacturing questions. While the MRL 8 assessment checklist has 14 fewer core questions on manufacturing at this juncture, the working group stated these core manufacturing questions are addressed earlier in the acquisition framework, which is reflective of commercial best practices where such manufacturing topics and discipline are addressed, in contrast to DOD’s current practice.

Draft Deskbook Explains MRL Application and Assessments

The draft MRL deskbook is a detailed instructional resource on how to apply MRLs and conduct assessments of manufacturing risk and readiness, such as how to structure and apply evaluations to a technology, component, manufacturing process, weapon system, or subsystem using the MRL definitions. It also demonstrates how assessments should be carried out at various phases by the managers of science and technology projects and technology demonstration projects intending to transition directly to the acquisition community, as well as acquisition program managers and the people involved in conducting assessments. According to the working group, MRLs can not only be used to improve how DOD manages and communicates manufacturing risk and readiness, but can also give decision makers and manager’s better visibility into program risks. For example, a variety of manufacturing status and risk evaluations have been performed for years as part of defense acquisition programs in a variety of forms—for example, production readiness reviews, manufacturing management/production capability reviews, etc. However, these structured and managed reviews do not use a uniform metric to measure and communicate manufacturing risk and readiness.

MRLs, when used in combination with technology readiness levels, are expected to address two key risk areas—immature product technologies and immature manufacturing capability. The draft deskbook says that it is common for manufacturing readiness to be paced by technology readiness or design stability, and that it is not until the product technology and product design are stable that manufacturing processes will be able to mature. MRLs can also be used to define manufacturing readiness and risk at the system or subsystem level. For these reasons, the MRL definitions were designed to include a target level of technology readiness as a prerequisite for each level of manufacturing readiness. Figure 3 shows the relationship of MRLs to system milestones and technology readiness levels in the defense acquisition life-cycle framework.

MRL Assessments Provide Basis for Identifying, Planning, and Managing Program Risks

MRL assessments are intended to leverage better manufacturing knowledge, enabling managers to be aware of problems or risks early in development, when they are easier to resolve and before significant investments are made. In turn, these risks can be addressed earlier in the life cycle when costs are lower. For example, the ability to transition technology smoothly and efficiently from the laboratories, onto the factory floor, and into the field is a critical enabler for evolutionary acquisition.

Assessments can be applied to a technology, manufacturing process, weapon system, or subsystem using the definitions as a standard. As part of the assessment, a comparison is made between the actual MRLs and the target MRL levels. The difference between the two identifies the risks and forms the basis for assisting managers to develop a plan—called a manufacturing maturation plan—to remove or reduce them. Risks should be identified throughout the life cycle and, when targets are not met, the plan updated to ensure the appropriate MRL will be achieved at the next decision point. The manufacturing maturation plan identifies manufacturing risks and provides a plan for mitigating each risk area throughout the duration of the technology or product-development program. The draft MRL deskbook says every assessment of manufacturing readiness should have an associated plan for areas where the MRL has not achieved its target level. The deskbook requires a manufacturing maturation plan to include the most essential items in planning for the maturity of an element of assessment that is below its target MRL. These include a statement of the problem that describes areas where manufacturing readiness falls short of the target MRLs, including key factors and driving issues, solution options and consequences of each option, and a maturation plan with a schedule and funding breakout. Other information should include the status of funding to execute the manufacturing plan and specific actions to be taken and by whom, and the MRL to be achieved and when it will be achieved.

MRL Pilot Programs Show Positive Benefits

Army and Air Force programs have pilot-tested MRLs on science and technology and some major acquisition programs in an effort to increase the manufacturing readiness and maturity to higher levels appropriate to the phase of development. Both services performed MRL assessments on selected pilot programs to address manufacturing risks and assess technology transition. The Army reports numerous benefits from the use of MRLs such as manufacturing efficiencies, improved labor utilization, and cost benefits. Similarly, the Air Force has used MRLs to manage its manufacturing risks associated with new technologies, yielding tangible benefits. While MRLs cannot take full credit for all benefits derived in the pilot programs, officials noted they are a good way to manage, mitigate, and communicate—between science and technology, acquisition, the user, and the system developer—readiness and risks early and throughout the acquisition process to avoid major consequences from manufacturing- related problems. These programs provide insight on how the acquisition community can utilize MRLs within weapon system programs.

Army

In 2004, the Army’s Aviation and Missile Research, Development and Engineering Center began applying MRLs to various technologies in concept development, including those technologies transitioning to engineering and manufacturing development. Officials stated that without cost and manufacturing readiness planning, science and technology programs face certain barriers to transition, resulting in: (1) high unit production cost caused by a focus on technology without regard to affordability; and (2) manufacturing problems caused by design complexity resulting in a technology that is not feasible to manufacture. For example, the Army has applied MRLs to many programs, including warfighter-protection materials, Micro-Electro-Mechanical Systems, embedded sensors, and helicopter cabin structures. The warfighter- protection program—the next generation of helmets and body gear— reported that it was able to reduce scrap by 60 percent and reduced touch labor by 20 to 40 percent. On programs where cost benefits could be roughly calculated, the Army believes that MRLs, among other improvement initiatives, contributed to the $426 million in benefits on seven programs. MRLs were also used as a metric in the Technology Transition Agreement to communicate manufacturing maturity and facilitate a smooth transition to the acquisition community.

Air Force

Air Force officials we met with discussed using MRLs to assess and identify gaps and understand risks in manufacturing maturity that would delay technology transition into an advanced systems development program or a fielded system upgrade. The Air Force has conducted several MRL assessments on advanced technology demonstrations and major defense acquisition programs, including the MQ-9 Reaper Unmanned Aircraft, Joint Strike Fighter, Advance Medium-Range Air-to-Air Missile, X-band thin radar array, and Sensor Hardening for Tactical Systems. Officials reported that the use of MRLs have contributed millions of dollars in cost avoidance, increased production rates, and has accelerated technology transition. For example, the Air Force reported realizing $65 million in savings by addressing problems with a costly manual drilling process. MRLs were used to raise new drilling technology from MRL 4 to MRL 9, achieving a unit-cost savings of $17,000 per aircraft from reduced tooling, manpower, floor space usage, and time.

Because of MRL assessment’s success on advanced technology programs, the Assistant Secretary of the Air Force for Acquisition directed the program office to perform MRL assessments on key MQ-9 Reaper manufacturing processes and technologies. The MQ-9 Reaper is an unmanned aerial vehicle designed to provide a ground attack capability during reconnaissance and surveillance missions. Officials stated that the MRL assessment results have (1) identified five areas that needed review prior to a milestone C production decision; (2) identified two risks to full- rate production—mitigations are in progress; and (3) provided evidence to support the contractor’s ability to meet the production goal of two aircraft per month. To ensure that manufacturing requirements are enforced, officials have developed policy for programs managers to assess manufacturing readiness at key decision points. To support that policy, the Air Force has developed training for integrated product teams to execute the manufacturing readiness assessments. Also in August 2009, the Air Force Institute of Technology established a Manufacturing Readiness Assessment course to provide training for the assessments within the Air Force and is currently open to all services and industry.

DOD’s Proposed MRLs Embody Many Best Practices of Leading Commercial Firms

To successfully develop and manufacture their products, the commercial firms we visited used a disciplined, gated process that emphasized manufacturing criteria early and throughout the product’s development. To measure manufacturing maturity, these firms developed processes that give manufacturing readiness and producibility primary importance throughout the product-development process, focusing on producing a product, not developing a technology. The goal is business profitability, and manufacturing maturity is important to this process from the earliest stages.

The best practices they employed were focused on gathering a sufficient amount of knowledge about their products’ producibility in order to lower manufacturing risks and included stringent manufacturing readiness criteria—to measure whether the product was mature enough to move forward in its development. In most respects, these criteria are similar to DOD’s proposed MRLs. For example, as with MRLs, commercial firms assess producibility at each gate using clearly defined manufacturing readiness criteria, gain knowledge about manufacturing early, demonstrate manufacturing processes in a production-relevant emphasize the importance of effective supply-chain management.

Essentially, commercial firms emphasize these criteria in order to maximize their understanding of manufacturing issues, to mitigate manufacturing risks that could affect business profitability or schedule goals for getting the product to market. DOD’s MRLs were designed to mitigate similar manufacturing risks. However, the difference is that the commercial firms we visited required that their manufacturing processes be in control prior to low-rate production, whereas DOD’s proposed MRL criteria do not require as early control of the manufacturing process.

DOD’s MRLs Are Similar to Manufacturing Criteria Used by Leading Firms

Leading commercial firms use manufacturing readiness criteria, similar to DOD’s MRLs, to assess the producibility of a system, gathering knowledge about the producibility of a product and the maturity of the manufacturing process. These criteria are applied early, even before a product formally enters into development, to identify and manage manufacturing risks and gaps. Additional manufacturing readiness criteria are applied through all the stages of a product’s development and production until the product is ready for commercial release. The firms we visited used manufacturing readiness criteria to measure both the readiness of the product or material to enter into development and to proceed through the necessary gates. Table 3 below shows examples of manufacturing readiness criteria that are common to both the MRLs and the commercial criteria, to illustrate their similarities. Both emphasized identifying risks and developing plans to mitigate these risks, setting realistic cost goals, and proving out manufacturing processes, material, and products.

Best Practice: Commercial Companies Emphasize Manufacturing Criteria Early and at Every Stage of the Product-Development Life Cycle

Each commercial firm we visited developed a disciplined framework for product development that assessed producibility at each gate using clearly defined manufacturing-maturity criteria that are similar in many respects to DOD’s MRLs. These include assessments of all aspects of manufacturing technology and risk, supply-chain issues, production facilities and tooling, and materials. Throughout the product-development life cycle, these criteria were applied to determine entry or exit into the next phase and led to informed decisions about whether the product was ready to move forward in its development. Manufacturing risks—such as those found in new manufacturing technologies or production facilities, new or revolutionary materials or supply-chain issues—were assessed at each step. Deliverables, including risk-identification and mitigation plans, manufacturing plans, and funding and resource needs, were required at each gate in order to progress to the next product-development gate. Targets were developed for each gate, including cost, schedule, and yield goals, and the product team was responsible for either meeting these targets or having risk-mitigation plans in place if the targets had not been met.

GE Aviation exemplifies this disciplined process, using a highly structured gated process with detailed checklists for entry and exit into each phase. Like DOD’s MRLs, these checklists contain increasingly detailed criteria— as they move from product start to production—for evaluating manufacturing technologies, cost drivers, materials, and supply-chain issues. Structured teams are brought together, tools are identified for execution and control of the process, and scheduled reviews are conducted with defined deliverables and checklists for each milestone. At each milestone, a vigorous review of the plans for the product’s development and manufacturing and risk-reduction efforts highlights issues before they become problems. The firm’s goal is to have mature processes by production. To achieve this, it considers manufacturing readiness throughout. Each project’s team is cross-functional and includes senior management, mid-management and the project team. This robust review process leverages expertise across GE Aviation, reduces risk, and highlights issues before they become problems.

As with all the commercial firms we visited, GE Aviation requires strong management involvement at each gate, along with decision reviews to determine if enough knowledge is available and risk-mitigation plans are in place to proceed or if actions to address and mitigate manufacturing risks can show a viable way forward. This allows management to resolve problems rather than pass them on to the next phase. At project start, which corresponds to MRL 4, the senior leadership team and product leadership team generate the product idea and assess the need for the project. They provide linkage between the business strategy and the project and develop the high-level project strategy. They identify any new product material or manufacturing processes and begin to develop a risk- reduction strategy for these issues. By the time the product enters the preliminary design phase, senior leadership and project teams agree on the approach to the project. At this time, product directors must have a manufacturing plan in place in order to identify how they are going to achieve manufacturing readiness. Technical risks are identified in the manufacturing plan, as well as risk-abatement strategies for materials and manufacturing processes and supply-chain risks. The plan has to show how issues will be successfully addressed by the detailed design phase, when leadership, the project team, and customers agree on the product to be delivered. If agreement is reached, they freeze the project plan and a decision is made to fund or terminate the project.

Multidisciplinary Team / Manufacturing Experts

In the commercial firms we visited, product-development teams were multidisciplinary, generally including management, manufacturing, quality, finance, suppliers, and engineering, with necessary skills available to assess manufacturing readiness. Leading firms recognize the value of having a knowledgeable, well-trained, and skilled manufacturing engineering workforce involved in these multidisciplinary teams from the beginning and throughout the process. When Honeywell reorganized its aerospace business in 2005, it created an advanced manufacturing engineering organization to focus on manufacturing concerns in the earliest phases of new product-development programs. This organization consists of engineers to support various manufacturing disciplines in Honeywell. An important part of this advanced engineering organization is its technology group, which consists of a select number of technology fellows with extensive expertise in key manufacturing disciplines that touch nearly all the products Honeywell produces. Honeywell retains highly skilled manufacturing expertise through this program and uses these experienced and knowledgeable manufacturing engineers to oversee each project’s manufacturing assessments.

Maturing Technology and Manufacturing Processes

Commercial firms focus on maturing and validating technology and manufacturing processes before these are associated with a product and before entry into the gated process. They keep invention and unproven technologies in the technology base until their producibility at the scale needed can be proven. As an example, GE Healthcare’s Gemstone scintillator underwent years of laboratory development on a small scale until GE Healthcare was satisfied that this material was ready to be used on its computed tomography (CT) scanners. Scintillators work by converting the X-rays in the CT scanner into visible light. GE Healthcare had been manufacturing its own scintillators since the late 1980s, but it needed an improved one that worked faster, for better clarity of vision and to reduce the amount of exposure to radiation. In 2001, the firm began basic composition development at the laboratory scale and narrowed down the alternatives to find the material with the best properties for this use. Even at this early stage, several years before the material would enter into GE Healthcare’s gated process, there was early engagement by the chemists with the manufacturing side. Before they decided on a solution, a determination was made that it could produce them with sufficient yield and quality: even if a material had the best optical qualities, it had to balance this with its producibility. GE Healthcare tested thousands of alternatives to determine what could meet its technical requirements and be producible in the quantities needed. The firm narrowed it down to a garnet-based, rare-earth minerals composite, and began producing it in small but increasing quantities. After narrowing the field to this garnet- based compound, GE Healthcare began to determine its suppliers and what equipment was needed. The firm then began building its first pilot plant to produce the material and the scintillators, 2 years before the scintillator entered the firm’s gated process. Figure 4 shows a photo of a CT scanner that uses the scintillator technology.

Best Practice: Commercial Firms Have Adopted DOD’s MRLs or Are Employing Similar Criteria in Their Product- Development Process

Because leading commercial firms focus on producibility as a key element to successfully develop products, they use rigorous analysis methods to assess producibility and to identify and manage manufacturing risks and gaps. They apply these methods and tools early and throughout product development and use them to manage their product development on a daily basis. This commercial approach is a process in which quality is designed into a product and manufacturing processes are brought into statistical control to reduce defects, in contrast to practices employed by many defense contractors where problems are identified and corrected after a product is produced.

Some firms were familiar with the DOD MRL proposal and had taken st to use the concepts at their own companies. Honeywell, for example, determined that early decisions were responsible for many production issues and so they developed analytical tools and models that support evaluations of manufacturing and risk throughout the product- development life cycle. In 2005, Honeywell engineers began looking for a way to measure manufacturing readiness and producibility, since they realized that early program decisions were driving many production iss and that by the time a product entered engineering and manufacturing development, it was too late to efficiently affect these issues. Some of these issues include cost overruns, quality problems, low-yield issues, service and maintainability inefficiencies, and supply-chain problems.

The output of this tool is an MRL assessment score that can identify gaps or risks. For example, spreadsheets show the MRL scoring at a glance for each of the elements evaluated, pinpointing the gaps; risk worksheets to quantify the risks; and action plans to close the gaps and mitigate these risks. It links to the firm’s gated process, providing entry and exit criteria and feedback on how to meet these criteria. The important information obtained is not necessarily what MRL level the item is at currently, but rather the robustness of the gap-closure plan to get to the desired level for the next gate. The application of the MRL tool helps identify what these key gaps are and what steps are required to close them.

The three enabling producibility tools that provide support for this assessment and early input on the producibility risks are: a Design for Manufacturing Model, a Product Complexity Model and a Yield Prediction Model: Manufacturing Complexity Model: This model identifies the design features that are driving manufacturing complexity into the design and enables scenarios to be evaluated to see what actions can be taken to simplify the design. Higher-complexity designs generally cost more and are higher risk, so the goal is to identify alternative design solutions that minimize complexity, but still meet all the performance requirements.

Yield Prediction Model: Honeywell has also developed yield prediction models based on statistical principles that correlate opportunities for defects in a design to established process capability benchmarks. This approach is used to predict yield during early design activities based on knowledge of the manufacturing processes used and the complexity of the design.

Design for Manufacturing Scorecard analysis: The third Honeywell- developed tool is a design for manufacturing scorecard, which quantifies how well the design adheres to recommended best practices. The goal of using the tool is to provide feedback to the designers so that they see how their design decisions directly affect producibility and help pinpoint improvement areas early in the process.

Honeywell then conducts an MRL workshop, with a team led by an engineer from its Advanced Manufacturing Engineering group that includes the program manager and various subject-matter experts. This team reviews the tools and the MRL criteria to gain consensus on ratings for each category. Honeywell’s Manufacturing Maturity Model, with input from these enabling tools, is used to develop an MRL score for the product. These assessments provide early producibility evaluations essential to mitigating design-driven risks. Since many producibility issues are driven by early design architecture decisions, these tools provide a way to analyze these decisions early and make the necessary performance and producibility trades through “virtual prototyping” long before actual hardware is built. The MRL score provides the necessary framework to ask the questions that such an analysis needs to answer.

After the MRL assessment is complete and the MRL scores and risk- mitigation plans are approved, the MRL analysis and risk mitigations are incorporated into the daily schedule of the program office. The office continually monitors the MRL levels, updating them and working toward its risk mitigation goals.

Best Practice: Leading Firms Prove Out Manufacturing Tooling, Equipment, and Processes before Entry into Production

Companies we visited spent years prior to production developing and proving out their manufacturing processes, including building test articles on pilot production facilities to perfect these processes. This allowed them to perfect and validate these processes, eliminate waste and scale up gradually to the required manufacturing level. They reduce errors and inefficiencies with the purpose of retiring manufacturing risks.

GE Aviation officials told us that certain advanced manufacturing technologies achieve significant cost savings by getting the costs lower earlier in the process and decreasing cycle time for faster implementation. An example of manufacturing techniques or processes that have made a big difference in costs, accuracy, and reliability include processes for drilling small shaped holes for turbine airfoils.

GE Aviation’s Turbine Airfoils Lean Lab provides a mock-up of a production facility or process, where such technologies and production processes can be tested to eliminate waste, scrap, and excess steps. They focus on one part family or process, such as the turbine airfoil shaped-hole manufacturing. The turbine airfoil is a part of the jet engine that generates power—it extracts horsepower from the high-temperature, high-speed combusted gasses. Turbine airfoil blades require hundreds of cooling holes that help maintain part integrity at elevated operating temperatures. Traditionally, round holes were used, but the technology has evolved to compound-angle-shaped holes, which improve cooling effectiveness and reduce engine stress. These type of holes cannot be economically produced by traditional methods and require improved manufacturing techniques. Advanced laser drilling was determined to be feasible, and GE Aviation decided to initiate the program through the Lean Lab to ensure manufacturing readiness of the process.

GE Aviation officials compared their processes in this case to DOD’s MRLs. Prior to entering their gated process, they began making investments in potential technologies, including tooling (MRL 1-3). As the gated process began, risks were identified and risk-abatement plans were put in place (MRL 4). GE Aviation then set up the Lean Lab to test the way the airfoil would actually be built. New processes were introduced that included new laser methods for hole drilling, improved robotic technology, machining, and grinding (MRL 5-6). The managers then ran the pilot production line for some time to manufacture these airfoils using actual production operators to be confident that the process would translate to the production line. Adjustments were made to improve efficiency and retested on the line until they were satisfied that they had worked out the best procedures. GE had tooling-design experts on the team at the Lean Lab to provide rapid part and tool manufacturing. Processes were brought into statistical control in order to take the complexity out of manufacturing, simplify the process, and reduce waste (MRL 7-8). They then dismantled the production line at the Lean Lab, took it to the manufacturing facility, and set it up exactly the same, with no variations allowed (MRL 9). This seamless introduction of the new manufacturing technology and the lean principles developed in the lab are expected to save many millions of dollars across GE Aviation, on production of this part family alone. Figure 6 shows a photo of GE Aviation’s Lean Lab setup.

GE Healthcare provides another example of proving out manufacturing processes prior to production in their development of the Gemstone scintillator for use on their CT Scanners. In 2003, the technology for this transitioned into the firm’s formal gated process or product start-up, and it began a detailed and extensive development of the manufacturing process. The firm built a pilot plant for this purpose and began manufacturing the composite in increasing amounts. In this first pilot plant, it was able to process the materials in increased quantities from what it produced in the lab. GE Healthcare verified that it had the right technologies to minimize manufacturing risks. In the laboratory environment, the firm had already answered the question “Can this composite be made with the desired properties?” and now asked “Can it be made with sufficient yield and quality to be manufactured in the desired amounts?” This early engagement with manufacturing enabled the firm to develop the process and reduce errors and inefficiencies with the purpose of reducing manufacturing risks.

GE Healthcare then built a second pilot production plant that further increased the amount produced above that of the first pilot plant. The firm continued its focus on gaining knowledge early, but on a larger scale: building the pilot plants was important to perfecting the process and gaining knowledge about the material’s producibility. At this stage, which coincides with MRL 8, it eliminated most of the technical risks involved in manufacturing the material. The firm then began to build its full-scale facility, which was ready 18 months before product launch.

When the full-scale production facility was completed, further scale-up of the material’s manufacturing became the focus. Changes to the design were made as needed to facilitate this. Any remaining manufacturing risks were eliminated prior to entry into the next stage, the product-validation stage. The Food and Drug Administration requires validation of finished medical devices. GE Healthcare told us that this means that all the equipment, processes, procedures, and factory workers are the same as will be used in actual production. Through use of the pilot plants to perfect the manufacturing of the scintillator material, GE Healthcare was able to produce production-representative material to satisfy this requirement.

Best Practice: Commercial Firms Work Closely with Suppliers, Who Must Meet High Quality Standards for Parts and Supplies

Commercial firms focus on developing strong relationships with their suppliers to ensure quality parts are provided in a timely manner. This begins with rigorous supplier-selection criteria to create a strong supplier base to provide quality parts. Similarly, DOD’s MRL supply-chain thread focuses on supplier capability throughout the acquisition life cycle, from as early as pre–milestone A (MRL 3), where initial assessment of the supply chain begins, through MRL 5, where supply-chain sources have been identified, and continuing to MRL 8, where the supply-chain should be stable and adequate to support low-rate production. Commercial firms generally have long-term relationships with these suppliers and can identify the supplier that is the best source of material or parts early, well before production begins. Leading commercial firms apply the same standards to these suppliers as they apply to their own manufacturing processes, such as ISO 9000 or other quality standards. Throughout product development and production, they establish effective communications with their suppliers so they can continually assess their performance. These firms work closely with their suppliers to retain these beneficial relationships, providing training where necessary and assistance if manufacturing problems arise.

GE Healthcare suppliers have to be validated before production begins, but qualifying them starts in the design phase. Suppliers are expected to meet the ISO 9000 standards and the Food and Drug Administration’s medical devices standards, but GE Healthcare’s own standards are more stringent that those. The supplier-qualification process ensures that suppliers meet GE Healthcare’s requirements, have a quality system that provides the appropriate controls for the part provided and meet regulations and requirements of multiple agencies, such as the Food and Drug Administration. Once a supplier is qualified, it becomes an approved supplier.

GE Healthcare also audits most of its suppliers and looks for issues such as lapsed ISO 9000 certification or a failed review. If it finds these things, GE Healthcare will ask the supplier for a plan to correct the deficiency and reaudit the supplier. GE Healthcare does annual risk assessments on the suppliers, based on data gathered during these audits, with sole-source or single-source suppliers being a high risk. If a supplier falls out of qualified status, GE Healthcare will do more frequent assessments. It constantly monitors the suppliers for quality. It helps the supplier get to the quality needed, but quality goals must be met.

Siemens is a global company that employs about 70,000 people in the United States. We visited Siemens Mobility Division, which builds light rail cars for public transit. Siemens places special emphasis on its supplier relationships, since it knows its suppliers can contract to other rail-car builders, as there is competition for suppliers in this market. If it has a good relationship with its suppliers, it can continue to benefit from the relationships with high-quality suppliers. Once it qualifies a supplier, it takes the responsibility for keeping the supplier qualified, providing technical assistance if necessary to keep the supplier in its pipeline. Even as early as the bid phase of the contract, Siemens knows who it will need as suppliers and if any particular supplier is new or challenged in some respect.

Siemens applies a three-step supplier-qualification process to its suppliers. This starts with a supplier self-assessment. The firm’s supplier- qualification personnel then visit the supplier’s plant and evaluate the supplier on the same self-assessment form, to determine if the supplier will make it to the vendor-qualification list. Once a supplier is on the approved vendor-qualification list, Siemens does risk ratings for these vendors to be sure it can keep them on the qualified-vendor list. The firm updates these assessments if the vendor situation changes, rating the vendor at low risk if it is fully qualified and working with it if some aspects are not qualified. Siemens takes responsibility for keeping the approved suppliers qualified, since finding and qualifying new vendors can be time- consuming and risky. It tries not to overload any one supplier, because some of their suppliers are small or specialty operations, so it keeps a pool of qualified suppliers for as many parts or materials as it can.

Commercial Firms Require That Manufacturing Processes Be in Control Earlier Than DOD’s MRLs

Although the firms we visited used manufacturing readiness criteria similar to DOD's proposed MRLs, one important difference we observed is that the commercial best practice is to have manufacturing processes in control prior to the production decision, while DOD's MRLs require manufacturing processes and procedures to be established and controlled during MRL 9, which occurs after the milestone C production decision, which authorizes a program to enter low-rate initial production, or equivalent. Although DOD’s MRLs incorporate many of the commercial manufacturing best practices into their manufacturing design and implementation criteria, the process controls criteria would be met too late in the process to achieve their full effect. DOD’s MRL matrix states that low-rate production yield and rate targets should be achieved at MRL 9, after the production decision has been made. The commercial firms we talked to emphasized that production processes must be in control before this decision is made. They realize that they are unable to make predictions about production performance until the process is stable and defects are predictable. Not achieving process control could result in low quality, extensive rework and waste, and not meeting cost and schedule targets. Firms established pilot lines to prove out production material, processes, and tooling, and worked to get processes under control before the system could move from the pilot line to production line. Figure 7 shows a depiction of the commercial manufacturing process approach.

The companies we visited used various approaches to build process capability and provide timely information on whether manufactured components, subsystems, or systems meet design specification. For example, GE Aviation uses a statistical measurement, called Z sigma level, to determine whether its processes have been brought under control or if variations in its manufacturing process could affect the quality of the product. The product is not moved into production until the firm is satisfied that these processes are in control. Similarly, GE Healthcare’s milestone process requires that a set of quality targets are part of the program and that those quality targets are met. Measures of process control vary from company to company, such as using yield or scrap and rework rates or sigma levels, but each looks carefully at those measures to ensure they carried no product-quality risk and uses this information to determine if the product is ready to be manufactured.

Two Successful DOD Programs Used Criteria Similar to Commercial Firms

Two DOD programs, the Army’s Lakota aircraft and the Missile Defense Agency’s Standard Missile 3 Block 1A, that had successful manufacturing outcomes employed some of the same practices as leading commercial firms. Both used a type of manufacturing readiness criteria to evaluate whether the programs were ready to enter into production and both programs focused on manufacturability as a key indicator of program success, using well-developed technology and a conservative approach in design and development.

The Lakota aircraft, a light utility helicopter that conducts noncombat missions, was a mature aircraft design when the Army entered into the contract with the European Aeronautic Defence and Space Company to purchase this commercially available helicopter. The program shows how careful attention to manufacturing readiness can reduce program risks. According to program office officials, the contractor was chosen in part because of its manufacturing track record, and it completed extensive planning, both internally and with its supplier base, to ensure on-time and reliable deliveries. Production planning and preparation were accomplished, including assessments of the manufacturing processes, capabilities, and facilities. These assessments determined that the program was low risk and ready for full-rate production. The Lakota is currently in full-rate production and has met its cost and schedule targets.

The Standard Missile 3 is a ship-based, antiballistic missile used by the Aegis ballistic missile defense system. Similar to the Lakota, the system met its cost and schedule goals by using an incremental, low-risk approach. Like the commercial firms we visited, the program built knowledge through the use of a type of manufacturing readiness criteria, which allowed the early identification of risk and implementation of mitigation strategies. The Standard Missile 3 Block IA was also on target for manufacturing cost and schedule and reported a lower cost per unit than was originally estimated on its production buys. As in the successful commercial firms we visited, manufacturing issues were considered very early in the design phase, leading to minimal changes in the program from flight test to production.

MRLs Are Hampered by Lack of an Agencywide Policy and Manufacturing Workforce Concerns

While acceptance of MRLs is growing within DOD and the defense industry, the services’ leadership appears to be resistant, and adoption efforts have been slow. For example, obtaining agreement on a policy that would institutionalize MRLs defensewide has proven difficult. Concerns raised by the military-service policymakers have centered on when and how the MRL assessments would be used. Officials responsible for the draft policy have promoted MRLs as an initiative that can address the manufacturing element in the design and production of weapon systems, citing commercial best practices that employ similar methods, and benefits derived from pilot programs. While extensive efforts have been made to promote the benefits of MRLs in support of a revised draft policy, it has taken nearly 2 years to allay concerns and it has not yet been approved. DOD is likely to face serious challenges even if an agreement is reached to approve the policy, however, because the number of DOD’s production and manufacturing career-field employees has diminished, particularly within the Air Force. Although the services are at the beginning stages of revitalizing their production and manufacturing workforce, DOD currently does not have adequate in-house expertise with the requisite knowledge to assess manufacturing throughout DOD. Essentially, the military services and Defense Contract Management Agency have identified knowledge and manpower gaps in their manufacturing workforce and believe that any initiative deploying MRLs defensewide could be hampered as a result.

Draft Policy to Institutionalize MRLs Has Proven Difficult, but the DOD Community Is Starting to See Its Value

While acceptance of MRLs is growing within DOD and the defense industry, the Army’s, Navy’s, and Air Force’s leadership appears to be resistant and adoption efforts have been slow. For example, a July 2008 draft MRL policy memorandum garnered disagreement among the military- service policymakers. The military services’ leadership agreed that MRLs provide value in the early acquisition phases but disagreed with the policy’s intent to formalize the process. For example, the MRL policy memorandum stated that on the basis of analyses by GAO and the Defense Science Board—as well as positive results on two Air Force pilot programs—that acquisition category I programs be assessed using the MRL scale. In particular, the draft policy included provisions that would require programs at milestone B to be assessed at MRL 6 or beyond for all critical technologies; programs at milestone C to be assessed at MRL 8 for all critical technologies; procedures to be coordinated for including assessments of manufacturing readiness in addition to technology readiness assessments at milestone B and C; and incorporation of guidance into training materials and guidebooks on best practices for addressing manufacturing from the earliest stages of development through production and sustainment.

In response to the draft policy, each of the military services issued memorandums in July 2008 to the Under Secretary of Defense (Acquisition, Technology and Logistics) or the Director, Defense Research and Engineering, stating they support MRLs and their use earlier in the acquisition process but they saw limited value in doing formal assessments prior to milestone C. In general, the services had concerns on when and how MRL assessments would be used. More specifically, their concerns included the following: Evaluation results that could be used as the basis for go / no go decisions.

A growing number of assessments being levied on acquisition programs.

Resources required to prove out multiple production lines in a competitive prototyping environment during the technology- development phase.

Since 2008, officials responsible for the draft policy memorandum have been working to address concerns raised by the services. According to the working group, most concerns pointed to a need to clarify how the information is intended to be used by decision makers at key milestones, particularly at the earlier milestones. According to the working group officials we interviewed, the intent is to inform decision makers with critical information—such as manufacturing risk and readiness measures, as appropriate to the phase of acquisition—so that knowledge-based decisions can be made earlier in the process to influence better outcomes in terms of cost and schedule in the later acquisition phases. Moreover, they cite that similar methods are employed by leading commercial firms as a best practice, plus the fact that MRL pilot programs have already demonstrated significant benefits. The revised MRL draft policy has not yet been approved. Officials familiar with the status of the draft policy stated that the leadership at one of the military services is still opposed to the idea of standardizing MRLs across DOD, and efforts to get approval have not yet occurred within the Office of the Director, Defense Research and Engineering.

DOD experienced similar problems introducing technology readiness levels. There was opposition to the use of technology readiness levels, but they became a standard for programs to follow, and the standard that technologies should be demonstrated in a relevant environment became a statutory requirement for all major acquisition programs seeking to enter system development. Programs report benefits from using technology readiness levels.

Some officials believe that MRLs could significantly reduce cost growth. For example, the Army and Air Force have reported MRLs were a factor that contributed to benefits of hundreds of millions of dollars in reduced program costs, improved schedule, and better performance of products.

MRL Acceptance Is Growing within DOD and Defense Industry

A number of Army, Air Force, and Missile Defense Agency programs—as well as defense contractors—have embraced MRLs as the method for assessing manufacturing maturity, risk, and readiness. For example, some Army commands have opted to use them on their science and technology efforts that have manufacturing elements, and have developed a formal process for identifying them. Similarly, two of three Air Force product centers under the materiel command—the Aeronautical Systems Center and the Air Armament Center—have recently issued local policy that mandate the use of MRLs. For example, in a policy memorandum by the Aeronautical Systems Center, dated October 13, 2009, all programs are now required to have manufacturing readiness assessments using MRLs, prior to each major milestone review. The memorandum acknowledged that the transition to production has historically been challenging for many programs and that manufacturing assessments are a key tool to ensure that programs are ready to begin production. The Missile Defense Agency has included MRLs as part of their assessment criteria. In addition, senior missile defense manufacturing personnel have developed and conducted training on how to conduct these assessments.

Similarly, a number of defense contractors have implemented MRLs as a discipline for identifying, managing, and communicating manufacturing risk and readiness. These contractors report a number of benefits using the MRLs, including reductions in program costs and improved production schedule. For example, in 2006, Raytheon participated in pilot MRL program assessments involving the Advanced Medium-Range Air-to-Air Missile and a portfolio of other programs and concluded the approach makes good business sense to lower risk. Raytheon claimed cost reductions of 30 percent or more could be achieved by using MRLs. Raytheon officials state that the combination of technology and manufacturing assessment processes changes the culture by driving a collaborative partnership between programs, design, and manufacturing engineering earlier in the product-development life cycle where maturity efforts can have the greatest effect on improving program affordability and predictability. As a result, Raytheon is deploying MRLs as a standard across the organization. Lockheed Martin is exploring ways to integrate MRLs within its existing review processes. As previously discussed, Honeywell adopted MRLs for use on both its defense and commercial products, and developed several models as an analysis-based approach to quantify their producibility risks.

Manufacturing Workforce Knowledge and Manpower Gaps May Impede Implementation of MRLs

The services are in the beginning stages of revitalizing their manufacturing workforce, largely in response to a February 2006 Defense Science Board task force report on “The Manufacturing Technology Program: A Key to Affordably Equipping the Future Force.” The report acknowledged that both the manufacturing expertise in the workforce and program funding have declined, thus eliminating much of the engineering and manufacturing talent across DOD and the industrial base. The report concluded that what was once a promising career field in the military services—with promotion paths, training, and professional development— has been systematically eliminated over the past few decades. Table 4 shows the decrease in the manufacturing career field across DOD from 2001 to 2007.

As indicated, DOD’s manufacturing career workforce trends show an overall decline, with the Army and Air Force having had the biggest declines at 14 percent and 37 percent, respectively. According to a DCMA official, the agency experienced about a 30 percent decrease during the same timeframe. An Army official responsible for workforce planning activities noted, however, there are no positions designated specifically for manufacturing, which make it difficult to determine the true career workforce numbers in this category. Fewer experts mean that fewer people at both the working level and in leadership positions understand the processes involved in developing and manufacturing defense systems and their importance in producing high-quality and reliable systems. Further, fewer people are capable of conducting production-readiness reviews, evaluating industry’s work on programs, and staying abreast of industry research and development. According to a recent study, of major concern is that recent estimates show 30 percent of the civilian manufacturing workforce—classified as production, quality, and manufacturing—are eligible for full retirement, and approximately 26 percent will become eligible for full retirement over the next 4 years. This means DOD will soon have an exodus of its manufacturing workforce and, accordingly, must plan for this eventuality.

Although the services are at the beginning stages of revitalizing their production and manufacturing workforce, program officials believe they currently do not have the in-house expertise with the requisite knowledge to assess manufacturing, if MRLs were to be mandated and deployed across DOD. For example, in interviews with career planning officials at the military services, most report that they have workforce challenges in manufacturing knowledge gaps or insufficient number of personnel to conduct the work, or both. The Defense Contract Management Agency reported similar manufacturing knowledge gaps due to a lack of focus in this area, but it now has new leadership in place and is establishing plans to address these deficiencies. Essentially, these knowledge deficiencies affect many areas, such as policy support for programs, the ability to develop an effective strategic plan and investment strategy for manufacturing technology, the ability to implement MRLs and conduct assessments, and the ability to effectively and affordably acquire high- quality weapon systems.

Conclusions

MRLs, resourced and used effectively, offer the potential for DOD to achieve substantial savings and efficiencies in developing and acquiring weapon systems. MRLs have been shown to work in reducing the cost and time for developing technologies and producing systems. Moreover, they have been shown to work on individual programs, and some Army commands and Air Force centers have adopted them. They are consistent with commercial best practices and have even been adopted by some defense firms. Yet, they have not been adopted DOD-wide. MRLs are being met with resistance similar to that experienced by technology readiness levels when they were first introduced. However, technology readiness levels are now widely accepted and used across DOD.

While MRLs represent a common body of knowledge and reflect many of the practices used by leading commercial companies, there is room for improvement. Criteria used for getting manufacturing processes under control are still not specific enough, allowing demonstration of controls to occur too late in the process—after the milestone C decision authorizing low-rate initial production—whereas commercial firms require that critical processes be in control earlier. While MRLs represent positive change, unless these criteria are strengthened at the time a production decision is made, DOD will have missed an opportunity to reduce the risk of continued cost growth on acquisition programs. Moreover, use of MRLs would be enhanced by the development of analytical tools, such as those used by Honeywell, to support MRL assessments.

A serious concern is that DOD’s in-house manufacturing workforce has been diminishing for decades and that, therefore, could hamper successful implementation of MRLs. Unless DOD develops long-range plans to build its in-house manufacturing workforce, it may not be able to realize the full potential of integrating manufacturing readiness levels into its processes.

Recommendations for Executive Action

To ensure that DOD is taking steps to strengthen and improve the producibility and manufacturing readiness of technologies, weapon systems, subsystems, or manufacturing processes, we recommend that the Secretary of Defense do the following: Require the assessment of manufacturing readiness across DOD programs using consistent MRL criteria as basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs.

Direct the Office of the Director, Defense Research and Engineering to examine strengthening the MRL criteria related to the process capability and control of critical components and/or interfaces prior to milestone C, or equivalent, for low-rate initial production decision.

Direct the Office of the Director, Defense Research and Engineering to assess the need for analytical models and tools to support MRL assessments.

Assess the adequacy of the manufacturing workforce knowledge and skills base across the military services and defense agencies and develop a plan to address current and future workforce gaps.

Agency Comments and Our Evaluation

DOD provided us written comments on a draft of this report. DOD partially concurred with our recommendation to require the assessment of manufacturing readiness across DOD programs using MRL criteria, and concurred with our other recommendations. Their comments can be found in appendix IV of this report.

In its comments, DOD partially concurred with the recommendation that DOD programs be required to assess manufacturing readiness using consistent MRL criteria as the basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs. DOD cites the Department of Defense Instruction 5000.02 as reflecting on manufacturing throughout the acquisition life cycle and, specifically, establishing a framework to continually assess and mitigate manufacturing risks. In its remarks, DOD states that the manufacturing readiness criteria will be tailored to programs and embedded into reviews and assessment templates, including systems engineering reviews, preliminary design reviews, and critical design reviews as well as acquisition phase exit criteria.

While we are encouraged by DOD’s plans to incorporate manufacturing readiness criteria into various assessments, we are concerned about the absence of any reference to MRLs, which identify specific benchmarks for each acquisition phase. It is unclear from DOD’s comments whether it intends to use a common definition of manufacturing readiness as acquisition phase exit criteria or whether the exit criteria will be decided on a case-by-case basis. While tailoring to individual programs is appropriate, tailoring must take place in the context of well-understood criteria for moving from phase to phase. A hallmark of the commercial programs we have looked at in this and other reviews is the reliance on disciplined processes for assessing readiness to proceed into more costly development and production phases. Firm criteria are needed to identify and address producibility and manufacturing risks on a timely basis, before they result in expensive production problems.

We also received technical comments from DOD, which have been addressed in the report, as appropriate.

We are sending copies of this report to the Secretary of Defense; the Secretary of the Army; Secretary of the Navy; Secretary of the Air Force; Director, Missile Defense Agency; Director, Defense Contract Management Agency; and Office of Management and Budget. In addition, the report will be made available at no charge on the GAO Web site at http://www.gao.gov.

If you, or your staff, have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. The major contributors are listed in appendix V.

Appendix I: Scope and Methodology

This report compares the Department of Defense (DOD) and its large prime contractors’ manufacturing practices with those of leading commercial companies—with a focus on improving the manufacturing of defense weapon systems. Specifically, we assessed (1) the manufacturing problems experienced by DOD, (2) how manufacturing readiness levels (MRLs) can address DOD’s manufacturing problems, (3) how proposed MRLs compare to manufacturing best practices of leading commercial companies, and (4) the challenges and barriers to implementing MRLs at DOD.

To identify the manufacturing problems experienced by DOD, we performed an aggregate analysis of DOD programs from our annual assessment database. We also conducted case studies of four programs with known cost and schedule problems to make observations on the types of problems DOD weapon systems may experience. The programs we reviewed, along with the prime contractors responsible for developing the systems, are Joint Air-to-Surface Standoff Missile, an air-to-surface missile funded by the Air Force and developed by Lockheed Martin; Exoatmospheric Kill Vehicle, a ballistic-missile interceptor funded by the Missile Defense Agency and developed by Raytheon; Electromagnetic Aircraft Launch System, a launch system for aircraft carriers funded by the Navy and developed by General Atomics; and H-1 helicopter upgrade, tactical utility and attack helicopters funded by the Navy and developed by Bell Helicopter.

To evaluate the four DOD weapon programs, we examined program documentation, such as acquisition decision memos and production readiness reviews, and held discussions with manufacturing and systems engineering officials from DOD program offices, the prime contractors, and the Defense Contract Management Agency. Based on the information gathered through interviews conducted and documentation synthesized, we identified commonalities among the case studies.

To determine how MRLs can address the manufacturing problems experienced by defense programs, we conducted interviews with officials from the Office of Secretary of Defense, Office of the Director, Defense Research & Engineering, Joint Defense Manufacturing Technology Panel working group, National Center for Advanced Technologies, National Defense Industrial Association, and Defense Acquisition University on their observations on MRLs. We also reviewed the MRL deskbook, matrix (risk areas), analyses, and training materials. We also conducted interviews with Army, Navy, and Air Force officials who were involved or familiar with the pilot tests of MRLs on various programs. The pilot programs we examined at the military services include the following Army—micro electro-mechanical systems inertial measurement unit, micro electro-mechanical systems safety arm, ferroelectric and micro electro-mechanical systems phase shifter, low-cost materials for improved protection, rotorcraft cabin floor structure, embedded sensors, and armor manufacturing; Air Force—MQ-9 Reaper, F-35 Joint Strike Fighter induct inlet, high- durability hot exhaust structures, F-135 Pratt & Whitney propulsion system, sensor hardening for tactical systems, and X-Band thin array radar; and Navy—P-8A aircraft.

To identify practices and criteria used by leading commercial companies that can be used to improve DOD’s manufacturing process, we selected and visited five companies based on several criteria: companies that (1) make products that are comparable to DOD in terms of complexity, (2) are recognized as leaders in developing manufacturing readiness criteria, or (3) have won awards for their manufacturing best practices, or a mix of the above. We met with these companies to discuss their product- development and manufacturing practices and the steps that they take to mitigate manufacturing risks, ensure manufacturing readiness, and improve supplier quality. We met with these companies to discuss their product-development life cycle and the methods and metrics they use to measure manufacturing maturity and producibility; manufacturing risk management; supplier management; and the key factors in the company’s successful manufacturing outcomes. We generalized much of the information due to the proprietary nature of the data relating to their manufacturing processes. Several companies provided data on specific processes or products that they agreed to allow us to include in this report. We reported on four of the five companies we visited. The five companies we visited include the following GE Aviation, a leading aerospace company, whose portfolio includes commercial engines and services, military engines and services, business and general aviation, engine components, and aviation systems. We met with manufacturing and quality officials in Cincinnati, Ohio, and discussed their manufacturing practices and manufacturing maturity metrics. We also toured their Lean Lab production facility and saw how these practices were applied.

GE Healthcare, which manufactures a range of products and services that includes medical imaging and information technologies and medical diagnostics. We met with manufacturing officials at their Milwaukee, Wisconsin, plant and discussed their manufacturing practices, including the development and manufacturing of their Gemstone scintillator for use on advanced CT scanners.

Honeywell Aerospace, a global provider of integrated avionics, engines, systems, and services for aircraft manufacturers, airlines, business and general aviation, and military and space operations. We met with manufacturing officials at their Phoenix, Arizona, facility and discussed their manufacturing maturity processes and the models and tools they used to assess this.

Siemens Mobility, a division of Siemens that develops and builds light rail cars for the North American market. We met with manufacturing and procurement officials at their Sacramento, California, manufacturing and assembly plant to discuss the manufacturing processes used in building their rail cars and their supplier management practices.

Toyota Motor Engineering and Manufacturing is responsible for Toyota’s engineering design, development and manufacturing activities in North America. We met with officials in their production engineering division in Erlanger, Kentucky, and also in their Toyota Technical Center located in Ann Arbor, Michigan, and discussed their vehicle development process and their methods for assuring supplier quality.

At each of the companies, we interviewed senior management officials knowledgeable about the manufacturing methods, techniques, and practices used throughout manufacturing and product development to ensure manufacturing maturity and producibility of their products. In particular, we discussed their (1) product development life cycle and the methods, metrics, and tools used to determine manufacturing maturity and producibility, (2) methods for identifying and mitigating risks in manufacturing a product, and (3) methods for supplier management to provide steady supply of quality parts.

In addition, we compared the practices of commercial firms to two major defense weapon systems known to be producing systems within cost and schedule goals and with successful manufacturing outcomes. To evaluate these two programs, we examined program documentation and held discussions with program and contracting officials. The two systems we reviewed, along with the prime contractors responsible for developing the systems, are Lakota aircraft, a light utility helicopter that conducts noncombat missions, funded by the Army and developed by the European Aeronautic Defence and Space Company; and Standard Missile 3 Block 1A, a ship-based antiballistic missile, funded by the Missile Defense Agency and developed by Raytheon.

To determine the challenges and barriers to MRL implementation efforts, we interviewed officials who were involved with the draft policy to standardize MRLs, as well as the military-service policy organizations that commented on the proposal. We also synthesized the information gathered at the various levels throughout the defense community to determine the issues surrounding MRLs as well as their merits. These DOD organizations include the Office of the Secretary of Defense, each of the military-service policy groups and program offices, the Office of the Director, Defense Research and Engineering, Systems and Software Engineering, and Joint Defense Manufacturing Technology Panel. To obtain an understanding of the workforce challenges in manufacturing, we reviewed selected documentation—such as Defense Science Board studies—and interviewed officials at each of the military services and Defense Contract Management Agency who were responsible for workforce planning activities and revitalization initiatives.

We conducted this performance audit from January 2009 to February 2010 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Manufacturing Readiness Level (MRL) Definitions

MRL 1—Basic Manufacturing Implications Identified This is the lowest level of manufacturing readiness. The focus is to address manufacturing shortfalls and opportunities needed to achieve program objectives. Basic research (i.e., budget activity 6.1 funds) begins in the form of studies.

This level is characterized by describing the application of new manufacturing concepts. Applied research (i.e., budget activity 6.2 funds) translates basic research into solutions for broadly defined military needs. Typically this level of readiness in the science and technology environment includes identification, paper studies, and analysis of material and process approaches. An understanding of manufacturing feasibility and risk is emerging.

MRL 3—Manufacturing Proof of Concept Developed This level begins the validation of the manufacturing concepts through analytical or laboratory experiments. This level of readiness is typical of technologies in the science and technology funding categories of Applied Research and Advanced Development (i.e., budget activity 6.3 funds). Materials or processes, or both, have been characterized for manufacturability and availability but further evaluation and demonstration is required. Experimental hardware models have been developed in a laboratory environment that may possess limited functionality.

MRL 4—Capability to Produce the Technology in a Laboratory Environment This level of readiness is typical for science and technology programs in the budget activity 6.2 and 6.3 categories and acts as exit criteria for the materiel solution analysis phase approaching a milestone A decision. Technologies should have matured to at least technology readiness level 4. This level indicates that the technologies are ready for the technology- development phase of acquisition. At this point, required investments, such as manufacturing technology development, have been identified. Processes to ensure manufacturability, producibility, and quality are in place and are sufficient to produce technology demonstrators. Manufacturing risks have been identified for prototype build, and mitigation plans are in place. Target cost objectives have been established and manufacturing cost drivers have been identified. Producibility assessments of design concepts have been completed. Key design performance parameters have been identified as well as any special tooling, facilities, material handling, and skills required.

MRL 5—Capability to Produce Prototype Components in a Production-Relevant Environment This level of maturity is typical of the midpoint in the technology- development phase of acquisition, or in the case of key technologies, near the midpoint of an advanced technology-demonstration project. Technologies should have matured to at least technology readiness level 5. The industrial base has been assessed to identify potential manufacturing sources. A manufacturing strategy has been refined and integrated with the risk-management plan. Identification of enabling/critical technologies and components is complete. Prototype materials, tooling and test equipment, as well as personnel skills, have been demonstrated on components in a production-relevant environment, but many manufacturing processes and procedures are still in development. Manufacturing technology development efforts have been initiated or are ongoing. Producibility assessments of key technologies and components are ongoing. A cost model has been constructed to assess projected manufacturing cost.

MRL 6—Capability to Produce a Prototype System or Subsystem in a Production-Relevant Environment This MRL is associated with readiness for a milestone B decision to initiate an acquisition program by entering into the engineering and manufacturing development phase of acquisition. Technologies should have matured to at least technology readiness level 6. It is normally seen as the level of manufacturing readiness that denotes completion of science and technology development and acceptance into a preliminary system design. An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined and characterized, but there are still significant engineering or design changes, or both, in the system itself. However, preliminary design of critical components has been completed and producibility assessments of key technologies are complete. Prototype materials, tooling and test equipment, as well as personnel skills have been demonstrated on systems or subsystems, or both, in a production-relevant environment. A cost analysis has been performed to assess projected manufacturing cost versus target cost objectives and the program has in place appropriate risk reduction to achieve cost requirements or establish a new baseline. This analysis should include design trades. Producibility considerations have shaped system-development plans. Industrial capabilities assessment for milestone B has been completed. Long-lead and key supply-chain elements have been identified. All subcontractors have been identified.

MRL 7—Capability to Produce Systems, Subsystems, or Components in a Production-Representative Environment This level of manufacturing readiness is typical for the midpoint of the engineering and manufacturing-development phase leading to the post- critical design review assessment. Technologies should be maturing to at least technology readiness level 7. System detailed design activity is underway. Material specifications have been approved and materials are available to meet the planned pilot-line build schedule. Manufacturing processes and procedures have been demonstrated in a production- representative environment. Detailed producibility trade studies and risk assessments are underway. The cost model has been updated with detailed designs, rolled up to system level, and tracked against allocated targets. Unit-cost reduction efforts have been prioritized and are underway. The supply chain and supplier quality assurance have been assessed and long-lead procurement plans are in place. Production tooling and test equipment design and development have been initiated.

MRL 8—Pilot-Line Capability Demonstrated; Ready to Begin Low- Rate Initial Production This level is associated with readiness for a milestone C decision, and entry into low-rate initial production. Technologies should have matured to at least technology readiness level 7. Detailed system design is essentially complete and sufficiently stable to enter low-rate production. All materials are available to meet the planned low-rate production schedule. Manufacturing and quality processes and procedures have been proven in a pilot-line environment and are under control and ready for low-rate production. Known producibility risks pose no significant challenges for low-rate production. The engineering cost model is driven by detailed design and has been validated with actual data. The Industrial Capability Assessment for milestone C has been completed and shows that the supply chain is established and stable.

MRL 9—Low-Rate Production Demonstrated; Capability in Place to Begin Full-Rate Production At this level, the system, component, or item has been previously produced, is in production, or has successfully achieved low-rate initial production. Technologies should have matured to at least technology readiness level 9. This level of readiness is normally associated with readiness for entry into full-rate production. All systems- engineering/design requirements should have been met such that there are minimal system changes. Major system design features are stable and have been proven in test and evaluation. Materials are available to meet planned rate production schedules. Manufacturing process capability in a low-rate production environment is at an appropriate quality level to meet design key-characteristic tolerances. Production risk monitoring is ongoing. Low- rate initial production cost targets have been met, with learning curves validated. The cost model has been developed for the full-rate production environment and reflects the effect of continuous improvement.

MRL 10—Full-Rate Production Demonstrated and Lean Production Practices in Place This is the highest level of production readiness. Technologies should have matured to at least technology readiness level 9. This level of manufacturing is normally associated with the production or sustainment phases of the acquisition life cycle. Engineering/design changes are few and generally limited to quality and cost improvements. System, components, or items are in full-rate production and meet all engineering, performance, quality, and reliability requirements. Manufacturing process capability is at the appropriate quality level. All materials, tooling, inspection and test equipment, facilities, and manpower are in place and have met full-rate production requirements. Rate production unit costs meet goals, and funding is sufficient for production at required rates. Lean practices are well established and continuous process improvements are ongoing.

Appendix III: Manufacturing Readiness Level (MRL) Threads and Subthreads (Risk Areas)

Appendix (MRL)

III: Manufacturing Readiness Level Threads and Subthreads (Risk Areas)

Potential sources identiﬁ ed for technol- ogy needs. Understand state of the art.

Industrial base capabilities surveyed and known; gaps/ risks identiﬁ ed for preferred concept, key technologies, components, and/or key processes.

Industrial base assessment initiated to identify poten- tial manufac- turing sources. Sole/ single/ foreign source vendors have been identiﬁ ed and planning has begun to minimize risks.

Industrial Capa- bilities Assess- ment (ICA) for Milestone (MS) B has been com- pleted. Indus- trial capability in place to support manufacturing of development articles. Plans to minimize sole/ foreign sources complete. Need for sole/single/ foreign sources justiﬁ ed. Poten- tial alternative sources identi- ﬁ ed.

Industrial capabil- ity to support production has been ana- lyzed. Sole/ single/ for- eign sources stability is assessed/ monitored. Developing potential alternate sources as necessary.

ICA for MS C has been completed. Industrial capability is in place to support Low Rate Initial Produc- tion (LRIP). Sources are available, multi- sourcing where cost- effective or necessary to mitigate risk.

Industrial capability is in place to support start of Full Rate Produc- tion (FRP).

Industrial capability supports FRP and is assessed to support modiﬁ ca- tions, upgrades, surge and other potential manufactur- ing require- ments.

Manu- facturing technology concepts identiﬁ ed through ex- periments/ models.

Manufactur- ing Science and Advanced Manufacturing Technology requirements identiﬁ ed.

Required manufacturing technology development efforts initiated as applicable.

Manufacturing technology efforts continu- ing. Required manufacturing technology development solutions dem- onstrated in a production-rele- vant environ- ment.

Manufactur- ing technol- ogy efforts continuing. Required manu- facturing technology development solutions demon- strated in a production- representative environment.

Primary manu- facturing technol- ogy efforts concluding and some improve- ment efforts continuing. Required manu- facturing technology solutions validated on a pilot line.

Manufactur- ing technol- ogy process improve- ment efforts initiated for FRP.

Manu- facturing technology continuous process im- provements ongoing.

Produc- ibility improve- ments imple- mented on system.

Should be assssed

experimen- tal results. completed. Results con- sidered in selection of preferred de- sign concepts and reﬂ ected in Technology Development Strategy (TDS) key com- ponents/ technologies. initiated as ap- propriate. On- going design trades consider manufactur- ing processes and industrial base capabil- ity constraints. Manufactur- ing processes assessed for capability to test and verify in production, and inﬂ uence on operations & support. technologies/ components completed. Results used to shape Acquisi- tion Strategy (AS), Systems Engineering Plan (SEP), Manufacturing and Produc- ibility plans, and planning for Engineering and Manufacturing Development (EMD) or tech- nology insertion programs. Preliminary design choices assessed against manufactur- ing processes and industrial base capabil- ity constraints. Producibility enhancement efforts (e.g., Design for Manufacturing (DFX)) initiated. and related manufactur- ing process capability completed. Producibility enhancement efforts (e.g., DFX) ongoing for optimized integrated system. Manufactur- ing processes reassessed as needed for capability to test and verify potential in- ﬂ uence on operations & support.

Known producibil- ity issues have been resolved and pose no signiﬁ - cant risk for LRIP. during LRIP. Producibility issues/ risks discovered in LRIP have been mitigated and pose no signiﬁ cant risk for FRP. produc- ibility im- provements ongoing. All modiﬁ ca- tions, upgrades, Diminishing Manufactur- ing Sources and Material Shortages (DMSMS), and other changes assessed for produc- ibility.

System al- located baseline established. System and subsystem preliminary design sufﬁ cient for EMD. All enabling/critical technologies/ components have been demonstrated. Preliminary de- sign KCs deﬁ ned.

Detailed design of product features and inter- faces is complete. All product data essential for system manu- facturing has been released.

Should be assssed

technical re- quirements evaluated.

Performance Parameter (KPPs) identiﬁ ed for preferred systems con- cept. System characteristics and measures to support required capa- bilities identi- ﬁ ed. Form, ﬁ t, and function constraints identiﬁ ed, and manufacturing capabilities identiﬁ ed for preferred sys- tems concepts. design Critical Characteristic (KCs) initiated. Product data required for prototype component manufacturing released. component manufactur- ing has been released. Potential KC risk issues have been identiﬁ ed and mitigation plan is in place.

Design change trafﬁ c does not sig- niﬁ cantly impact LRIP. KCs are attain- able based upon pilot line demonstra- tions.

Conﬁ gura- tion Audit (PCA) or equivalent complete as necessary. Design change traf- ﬁ c is limited. All KCs are controlled in LRIP to appropri- ate quality levels. obsoles- cence. All KCs are controlled in FRP to appropri- ate quality levels.

Initial cost targets and risks identiﬁ ed. High level process chart model developed. Technology cost models developed for new process steps and materials based on experi- ments.

Manufacturing, material and special require- ment cost drivers identi- ﬁ ed. Detailed process chart cost models driven by pro- cess variables. Cost driver uncertainty quantiﬁ ed.

Prototype components produced in a produc- tion relevant environment, or simulations drive end- to-end cost models. Cost model includes materials, labor, equip- ment, tooling/ (Special Test Equipment (STE), setup, yield/ scrap/ rework, Work in Process (WIP), and capabil- ity/capacity constraints).

Cost model updated with design require- ments, material speciﬁ cations, tolerances, inte- grated master schedule, results of system/ subsystem simulations and production rel- evant prototype demonstrations.

Cost model updated with the results of systems/ sub-systems produced in a production- representative environment and with pro- duction plant layout and design and obsolescence solutions.

Cost models updated with results of pilot line build.

FRP cost model up- dated with result of LRIP build.

Cost model validated against actual FRP cost.

Should be assssed

Sensitiv- ity analysis conducted to deﬁ ne cost drivers and produc- tion devel- opment strategy (i.e., lab to pilot to factory).

Producibility cost risks as- sessed. Initial cost models support Analy- sis of Alterna- tives (AoA) and Alternative Systems Review (ASR).

Costs analyzed using prototype component ac- tuals to ensure target costs are achievable. Decisions re- garding design choices, make/ buy, capacity, process capa- bility, sources, quality, KCs, yield/rate, and variabil- ity inﬂ uenced by cost models.

Costs analyzed using prototype system/subsys- tem actuals to ensure target costs are achiev- able. Allocate cost targets to subsystems. Cost reduction and avoid- ance strategies developed.

Manufactur- ing costs rolled up to system/sub- system level and tracked against tar- gets. Detailed trade studies and engineer- ing change requests supported by cost esti- mates. Cost reduction and avoidance strategies underway.

Costs analyzed using pilot line actuals to ensure target costs are achievable. Manufac- turing cost analysis supports proposed changes to require- ments or conﬁ gura- tion. Cost reduction initiatives ongoing.

LRIP cost goals met and learn- ing curve analyzed with actual data. Cost reduction initiatives ongoing. Touch labor efﬁ ciency analyzed to meet production rates and elements of inef- ﬁ ciency are identiﬁ ed with plans in place for reduction.

FRP cost goals met. Cost reduction initiatives ongoing.

Program/ projects have reasonable budget estimates for reaching MRL 4 by MS A.

Manufacturing technology initiatives iden- tiﬁ ed to reduce costs. Program has reasonable budget esti- mate for reach- ing MRL 6 by MS B. Estimate includes capi- tal investment for production- relevant equipment. All outstand- ing MRL 4 risk areas understood, with approved mitigation plans in place.

Program has updated bud- get estimate for reaching MRL 6 by MS B. All outstand- ing MRL 5 risk areas understood, with approved mitigation plans in place.

Program has reasonable budget estimate for reaching MRL 8 by MS C. Estimate includes capital investment for production- representative equipment by CDR and pilot line equipment by MS C. All outstanding MRL 6 risk areas understood, with approved mitigation plans in place.

Program has updated budget estimate for reaching MRL 8 by MS C. All outstanding MRL 7 risk areas under- stood, with approved mitigation plans in place.

Program has reasonable budget estimate for reach- ing MRL 9 by the FRP decision point. Estimate includes investment for LRIP and FRP. All outstand- ing MRL 8 risk areas under- stood, with approved mitigation plans in place.

Program has reasonable budget estimate for FRP. All outstanding MRL 9 risk areas under- stood, with approved mitigation plans in place.

Production budgets sufﬁ cient for production at required rates and schedule to support funded program.

Should be assssed

Material properties validated and as- sessed for basic manufactur- ability using experi- ments.

Survey deter- mines that the projected material has been produced in a laboratory environment. Survey deter- mines that the projected material has been produced in a laboratory environment.

Materials have been manufactured or produced in a prototype environment (maybe in a similar applica- tion/program). Maturity efforts in place to address new material pro- duction risks for technology demonstration.

Mate- rial maturity veriﬁ ed through technology demonstration articles. Prelimi- nary material speciﬁ cations in place and material proper- ties have been adequately characterized.

Material maturity suf- ﬁ cient for pilot line build. Material speciﬁ cations approved.Ma- terial maturity sufﬁ cient for pilot line build. Material speciﬁ cations approved.

Materials proven and validated during EMD as adequate to sup- port LRIP. Material speciﬁ ca- tion stable.

Material is proven and controlled to speciﬁ ca- tion in LRIP. Material is proven and controlled to speciﬁ ca- tion in LRIP.

Material is proven and controlled to speciﬁ ca- tion in FRP. Material is proven and controlled to speciﬁ ca- tion in FRP.

Material scale-up issues identiﬁ ed.

Projected lead times have been identiﬁ ed for all difﬁ cult- to-obtain, difﬁ cult-to- process, or hazardous materials. Quantities and lead times estimated.

Availability is- sues addressed for prototype build. Signiﬁ - cant material risks identiﬁ ed for all materi- als. Planning has begun to address scale-up issues.

Availability issues addressed to meet EMD build. Long-lead items identiﬁ ed. Potential obso- lescence issues identiﬁ ed.

Availabil- ity issues addressed to meet EMD builds. Long-lead procurement identiﬁ ed/ planned for LRIP. Obsoles- cence plan in place.

Long-lead procure- ment initiated for LRIP. Availability issues pose no signiﬁ - cant risk for LRIP.

Long-lead procure- ment initi- ated for FRP. Availability issues pose no signiﬁ - cant risk for FRP.

Program is in FRP, with no signiﬁ cant material availability issues.

Initial as- sessment of potential supply chain capa- bility.

Survey completed for potential supply chain sources.

Potential supply chain sources identi- ﬁ ed.

Supply chain plans in place (e.g. teaming agreements and so forth) leading to an EMD con- tract award.

Effective supply chain management process in place. Assess- ment of criti- cal ﬁ rst tier supply chain completed.

Supply chain ade- quate to support LRIP. As- sessment of critical second and lower tier supply chain completed.

Supply chain is stable and adequate to support FRP. Long-term agreements in place where practical.

Supply chain proven and supports FRP require- ments.

Special handling procedures applied in pilot line environ- ment.

Special handling procedures effectively implement- ed in FRP.

Should be assssed

applied in the lab. Special handling concerns assessed. handling requirements identiﬁ ed. handling requirement gaps identiﬁ ed. New special handling pro- cesses demon- strated in lab environment. requirement gaps complete. ment. Special handling procedures developed and anno- tated on work instructions.

Special handling procedures demon- strated in EMD or technology insertion programs. Special han- dling issues pose no signiﬁ cant risk for LRIP. All work instructions contain special handling provisions, as required. handling procedures demonstrat- ed in LRIP. Special han- dling issues pose no signiﬁ cant risk for FRP.

Identiﬁ ca- tion of proposed manu- facturing concepts or produc- ibility needs based on high-level process ﬂ owchart models.

Production modeling and simulation approaches for process or product are identiﬁ ed.

Initial simula- tion models (product or process) de- veloped at the component level.

Initial simulation models devel- oped at the subsystem or system level.

Simulation models used to determine system constraints and identify improvement opportunities.

Simulation models veri- ﬁ ed by pilot line build. Results used to improve process and determine that LRIP require- ments can be met.

Simulation model veri- ﬁ ed by LRIP build, assists in manage- ment of LRIP and determines that FRP re- quirements can be met.

Simula- tion model veriﬁ ed by FRP build. Production simulation models used as a tool to assist in manage- ment of FRP.

Document high-level manufac- turing processes. Critical manufactur- ing process- es identiﬁ ed through experimen- tation.

Complete a survey to determine the current state of critical processes.

Maturity has been assessed on similar processes in produc- tion. Process capability requirements have been identiﬁ ed for pilot line, LRIP and FRP.

Manufacturing processes dem- onstrated in pro- duction-relevant environment. Begin collecting or estimating process capabil- ity data from prototype build.

Manufactur- ing processes demon- strated in a production- representative environment. Continue collecting or estimating process capa- bility data.

Manu- facturing processes veriﬁ ed for LRIP on a pilot line. Process capability data from pilot line meets target.

Should be assssed

to show FRP impact and potential for continuous improve- ment.

FRP objec- tives.

Initial estimates of yields and rates based on experi- ments or state of the art.

Yield and rates assessment on proposed/simi- lar processes complete and applied within AoA.

Target yields and rates established for pilot line, LRIP, and FRP. Yield and rate issues identiﬁ ed. Improvement plans devel- oped/initiated.

Yields and rates from production-rele- vant environ- ment evaluated against targets and the results feed improve- ment plan.

Yields and rates from production- representative environment evaluated against pilot line targets and the results feed improvement plans.

Pilot line targets achieved. Yields and rates required to begin LRIP veriﬁ ed using pilot line articles. Improve- ment plans ongo- ing and updated.

LRIP yield and rate targets achieved. Yield im- provements ongoing.

FRP yield and rate targets achieved. Yield im- provements ongoing.

Quality strate- gy identiﬁ ed as part of the TDS and included in SEP.

Quality strat- egy updated to reﬂ ect KC identiﬁ cation activities.

Initial quality plan and quality management system is in place. Quality risks and metrics have been identiﬁ ed. -lauQ ity targets established. Demonstrate ability to collect and analyze quality data (process and system) in the production- representative environment.

Qual- ity targets demon- strated on pilot line. Continuous quality im- provement ongoing. Supplier products have com- pleted qualiﬁ - cation testing and ﬁ rst-article inspection. Supplier products pass accep- tance test- ing at a rate adequate to begin LRIP.

Quality tar- gets veriﬁ ed on LRIP line. Continu- ous quality improvement ongoing. Supplier products pass accep- tance testing at a rate adequate to transition to FRP.

Quality tar- gets veriﬁ ed on FRP line. Continuous quality im- provement ongoing.

Should be assssed

New manufactur- ing skills identiﬁ ed.

Manufactur- ing skill sets identiﬁ ed and production workforce requirements (technical and operational) evaluated as part of AoA. Determine availabil- ity of process development workforce for the TDP.

Skill sets identiﬁ ed and plans devel- oped to meet prototype and produc- tion needs. Special skills certiﬁ cation and training requirements established.

Manufacturing workforce skills available for production in a relevant envi- ronment. Iden- tify resources (quantities and skill sets) and develop initial plans to achieve requirements for pilot line and production.

Manufactur- ing workforce resource requirements identiﬁ ed for pilot line. Plans developed to achieve pilot line require- ments. Plans updated to achieve LRIP workforce requirements. Pilot line workforce trained on representative environment.

Manu- facturing workforce resource re- quirements identiﬁ ed for LRIP. Plans developed to achieve LRIP re- quirements. Plans updated to achieve FRP workforce require- ments. LRIP personnel trained on pilot line where pos- sible.

LRIP personnel requirements met. Imple- ment plan to achieve FRP workforce re- quirements.

FRP person- nel require- ments met. Production workforce skill sets maintained due to attrition of workforce.

Tooling/STE/ SIE require- ments are considered as part of AoA.

Identify tool- ing and Special Test Equip- ment / Special Inspection Equipment (STE/SIE) requirements and provide supporting rationale and schedule.

Prototype tool- ing and STE/SIE concepts dem- onstrated in pro- duction relevant environment. Production tool- ing and STE/SIE requirements developed. noitcudorP tooling and STE/SIE design and development efforts under- way. Manu- facturing equipment maintenance strategy developed.

All tooling, test, and inspection equipment proven on pilot line and re- quirements identiﬁ ed for LRIP. Manu- facturing equipment mainte- nance demon- strated on pilot line.

All tooling, test, and inspection equipment proven in LRIP and requirements identiﬁ ed for FRP. Manu- facturing equipment maintenance schedule demon- strated.

Proven tool- ing, test, and inspection equipment in place to support maximum FRP. Planned equipment mainte- nance schedule achieved.

Specialized facility re- quirements/ needs identiﬁ ed.

Manufacturing facilities identi- ﬁ ed and plans developed to produce prototypes.

Manufacturing facilities identi- ﬁ ed and plans developed to produce pilot line build.

Should be assssed

evaluated as part of AoA. produce LRIP build. facilities adequate to begin LRIP. Plans in place to support transition to FRP.

LRIP. Capac- ity plans adequate to support FRP. maximum FRP require- ments.

Manufactur- ing strategy developed and integrated with acquisi- tion strat- egy. Prototype schedule risk mitigation efforts incor- porated into TDS.

Manufactur- ing strategy reﬁ ned based upon preferred concept. Proto- type schedule risk mitigation efforts initi- ated.

Initial manufac- turing approach developed. All system- design-related manufacturing events included in Integrated Master Plan/ Integrated Master Schedule (IMP/IMS). Manufacturing risk mitigation approach for pilot line or technology insertion pro- grams deﬁ ned. -unam laitinI facturing plan developed. Manufactur- ing planning required to achieve MRL 8 has been included in the IMP/IMS. Manufactur- ing risks integrated into risk miti- gation plans. Develop initial work instructions. Effective production control sys- tem in place to support pilot line.

Manufac- turing plan updated for LRIP. All key manufac- turing risks are identi- ﬁ ed and as- sessed with approved mitigation plans in place. Work instructions ﬁ nalized. Effective production control system in place to support LRIP.

Manufactur- ing plan updated for FRP. All manufactur- ing risks tracked and mitigated. Effective production control sys- tem in place to support FRP.

All manufac- turing risks mitigated.

Technology development article component list developed with associ- ated lead-time estimates.

Technology development part list matur- ing. Make/ buy evalua- tions begin and include production considerations reﬂ ecting pilot line, LRIP, and FRP needs. Lead times and other risks identiﬁ ed.

Most material decisions com- plete (make/ buy), material risks identiﬁ ed, and mitigation plans devel- oped. Bill of Materials (BOM) initiated. yub/ekaM decisions and BOM complete for pilot line build. Mate- rial planning systems in place for pilot line build.

Make/buy decisions and BOM complete to sup- port LRIP. Material planning systems in place for LRIP build.

Make/buy decisions and BOM complete to support FRP. Material planning sys- tems in place for FRP.

Mate- rial planning systems validated on FRP build.

Appendix IV: Comments from the Department of Defense

DoD Response to GAO-10-439 Recommendations GAO DRAFT REPORT DATED MARCH 12, 2010 GAO-10-439 (GAO CODE 120793) “BEST PRACTICE: DOD CAN ACHIEVE BETTER OUTCOMES BY STANDARDIZING THE WAY MANUFACTURING RISKS ARE MANAGED”

DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATION 1: The GAO recommends that the Secretary of Defense require the assessment of manufacturing readiness across DoD programs using consistent MRL criteria as a basis for measuring, assessing, reporting, and communicating manufacturing readiness and risk on science and technology transition projects and acquisition programs.

DOD RESPONSE: Partially concur: The Department of Defense recognizes that mature manufacturing processes and readiness are critical to achieving predictable and successful program outcomes. It also recognizes the value in assessing manufacturing risks during science and technology research on technologies planned to be incorporated into acquisition programs. Department of Defense Instruction (DoDI) 5000.02, Operation of the Defense Acquisition System, dated 8 December 2008 reflects an increased focus on manufacturing throughout the acquisition lifecycle for programs of all acquisition categories. Specifically, it establishes a framework to continually assess and mitigate manufacturing risks during the Analysis of Alternatives, 2366b certifications to Congress, Preliminary and Critical Design Reviews; and acquisition milestones.

The Department’s new manufacturing readiness criteria will form the basis for assessing pertinent science and technology efforts, and acquisition programs throughout the acquisition lifecycle on programs of all acquisition categories. These criteria will be a tool to identify relevant manufacturing risks which require mitigation. These manufacturing readiness criteria are expected to be tailored for programs and will be included in the Department’s criteria for systems engineering technical reviews; the Department’s templates for Preliminary Design Review/Critical Design Review reports; and acquisition phase exit criteria. These manufacturing readiness criteria will also be assessed as part of the Program Support Reviews which the Department conducts on Major Defense Acquisition Programs. These reviews evaluate manufacturing as part of an overall integrated program assessment. These manufacturing readiness criteria and products will be made available to government and industry. Their use by the Services on lower ACAT programs will also be encouraged. The Navy’s Gate Review process currently assesses manufacturing risks but is being updated with the new manufacturing readiness criteria.

DoD Response to GAO-10-439 Recommendations RECOMMENDATION 2: The GAO recommends that the Secretary of Defense direct the Office of the Director, Defense Research and Engineering to examine strengthening the MRL criteria related to the process capability and control of critical components and/or interfaces prior to the Milestone C low rate initial production decision.

DOD RESPONSE: Concur. Department of Defense Instruction 5000.02 directs that programs at Milestone C have no significant manufacturing risks; that manufacturing processes have been effectively demonstrated in a pilot line environment; and manufacturing processes are under control (if Milestone C is full-rate production). While the Department notes that all manufacturing processes do not warrant the same level of process capability and control, appropriate levels of control are certainly warranted on a case by case basis.

The Department will examine strengthening the manufacturing readiness criteria related to process capability and control of critical components and/or interfaces prior to the Milestone C low rate initial production decision. However, program offices and contractors should continue to have the latitude to jointly agree on the targets and specific process control demonstrations required on the pilot production line during the Engineering and Manufacturing Development to ensure success.

RECOMMENDATION 3: The GAO recommends that the Secretary of Defense direct the Office of the Director, Defense Research and Engineering to assess the need for analytical models and tools to support MRL assessments.

DOD RESPONSE: Concur. The Department will collaborate with government services, contractors, and academia to capture knowledge and provide improved tools for government and contractor usage in conducting assessments of manufacturing readiness as part of systems engineering technical reviews and milestone reviews.

RECOMMENDATION 4: The GAO recommends that the Secretary of Defense assess the adequacy of the manufacturing workforce knowledge and skills base across the military services and defense agencies and develop a plan to address current and future workforce gaps DOD RESPONSE: Concur. We agree that the production, quality and manufacturing (PQM) career field has suffered erosion, as have other DoD career fields. The USD (AT&L) Director of Human Capital has launched a review of the PQM career field design to identify the skills, knowledge and training required at each level of career progression in order to develop training courses and evaluate progression of anticipated DoD planned new hires. The Department has started to implement hiring and retention strategies to mitigate the potential loss in experienced, senior-level PQM talent and increase the size of the manufacturing workforce. As part of the Secretary’s growth strategy and other initiatives, the PQM career field is projected to grow approximately DoD Response to GAO-10-439 Recommendations 1,300 (13%) by FY2015. Each of the military services and other DOD components has been actively planning and deploying initiatives that support the DOD acquisition workforce growth strategy. Components have submitted planning inputs to OSD and to the Defense Acquisition Workforce Senior Steering Board, and growth is underway.

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Key contributors to this report were Karen Zuckerstein, Assistant Director; John M. Ortiz, Jr.; Beverly Breen; Leigh Ann Nally; Dr. W. Kendal Roberts; Andrea Bivens; Kristine Hassinger; Kenneth Patton; Bob Swierczek; and Dr. Timothy Persons, Chief Scientist.

Related GAO Products

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-10-388SP. Washington, D.C.: March 30, 2010.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-09-326SP. Washington, D.C.: March 30, 2009.

Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD’s Acquisition Environment and Weapon System Quality. GAO-08-294. Washington, D.C.: February 1, 2008.

Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes. GAO-06-883. Washington, D.C., September 14, 2006.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.

Why Some Weapon Systems Encounter Production Problems While Others Do Not: Six Case Studies. GAO/NSIAD-85-34. Washington, D.C.: May 24, 1985.