Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD
Published: 1996-08-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed the Department of Defense's (DOD) quality assurance practices, focusing on: (1) the problems that DOD has had in improving such practices; (2) private-sector practices that could be beneficial to DOD; and (3) DOD efforts to improve its quality assurance activities.
section_title: What GAO Found
paragraphs: GAO found that: (1) DOD spends more than $1.5 billion annually to support its quality assurance activities; (2) manufacturing and quality problems have contributed to cost and schedule overruns on DOD weapons systems programs; (3) DOD acquisition programs have experienced quality problems during the production phase due to incomplete weapon designs; (4) DOD is relying on unstable designs and inspections to rework the defects in its B-2 bomber and C-17 Airlifter programs; (5) a number of successful commercial manufacturers are improving the quality of their products, while reducing their oversight and inspection costs; (6) commercial manufacturers have broadened their definition of quality assurance, by changing it from a postmanufacturing step done at the end of each process to being part of the process itself; (7) DOD began in the early 1990s to implement a method of moving away from inspection-oriented quality assurance practices and toward process control; (8) DOD is developing a plan to reduce the costs of redundant quality assurance processes; (9) the plan will allow DOD contractors to use a single quality management system based on process controls for all DOD contracts; and (10) DOD could enhance its quality management system by encouraging defense contractors to use more advanced commercial techniques and quality assurance practices.

section_title: Letter
paragraphs: The Department of Defense (DOD) spends about $1.5 billion extra per year on military-unique quality assurance requirements for major acquisitions and billions more on cost and schedule overruns to correct problems caused partly by poor quality practices. To help improve DOD’s quality assurance program, we reviewed world-class commercial organizations to determine what practices they had adopted to more efficiently produce quality products. Specifically, this report describes (1) the historical problems DOD has had in improving quality assurance practices, (2) some private sector practices that could benefit DOD, and (3) a current plan for improving quality assurance activities.
subsections: section_title: Background
paragraphs: Quality assurance has a simple goal: to ensure that products perform the way they are supposed to. For many years, the traditional way DOD and commercial companies achieved quality was through systematic final inspection. But now, intense competition has led some U.S. companies to adopt total quality management practices that are prevention based. Consequently, quality assurance has taken on a broader meaning, to include virtually all key design and engineering elements during development, the transition to production, and production itself.
There is general agreement across government and industry that DOD’s inspection-based quality assurance practices have added unnecessary costs to acquisitions because they require DOD and contractor personnel and resources for oversight that are separate from the production process. Until recently, DOD’s quality requirements were based on MIL-Q-9858A, a military standard established in 1963. This standard requires a contractor to establish a quality program with documented procedures and processes that are subject to approval by government representatives throughout all areas of contract performance. Quality is theoretically ensured by requiring both the contractor and the government to monitor and inspect products. In June 1994, the Secretary of Defense announced that commercial quality standards such as ISO-9000 should replace MIL-Q-9858A where it makes sense. DOD will not require MIL-Q-9858A on contracts awarded after October 1996. Until then, DOD will be approving the quality standards on a case-by-case basis at the request of industry.
The Defense Contract Management Command (DCMC) has primary responsibility for setting and overseeing quality assurance standards within DOD. As of September 1995, DCMC had about 5,000 quality specialists located in contractor facilities or in regional offices across the country.
section_title: Changing DOD’s Quality Assurance Practices Is Challenging Given Long-standing Problems
paragraphs: DOD faces a formidable challenge in changing its quality assurance culture. This culture has been characterized by a narrow approach to quality assurance, in both DOD and the defense industry, which has led to a focus on detecting defects and recording corrective actions. In the past, DOD’s practices have reflected a narrow approach to quality assurance. Responsibility for meeting military quality standard MIL-Q-9858A has been assigned to one organization—DCMC, which is outside much of the acquisition decision-making process. DCMC has quality specialists stationed in contractor plants across the country to inspect material and end items. These specialists have relied heavily on quality system documentation, various reviews and audits of the quality process, and corrective action plans submitted by contractor quality assurance personnel to ensure quality on weapon systems.
Based on information from studies performed for the Secretary of Defense, we estimate that the extra cost associated with military-unique quality assurance requirements for DOD acquisitions is $1.5 billion annually. The studies assessed the cost impact of DOD regulations on contractors and DOD and found that the contractors’ cost to implement MIL-Q-9858A and to comply with DOD requirements represented at least 1.7 percent of DOD’s acquisition cost, or about $1 billion. Most of this cost occurred as the result of contractor quality assurance and operations personnel devoting time to such activities as preparing quality plans and procedures, conducting and documenting inspections, documenting deviations, proposing corrective actions to government concerns, and supporting government audits and reviews. Further, DOD’s own costs for quality assurance oversight were about $687 million annually. This estimate does not include what DOD has spent to correct the manufacturing and quality problems that have contributed to historical cost and schedule overruns on weapon system production programs. These past problems occurred partly because manufacturing processes were not considered during the design phase of the program, and inspection, rather than process control, was the predominant method of ensuring quality.
Like many commercial manufacturers, DOD is now being expected to cut costs and do more with less. Commercial manufacturers have adopted far-reaching quality strategies as one way to become more competitive, efficient, and economical. Even though DOD faces the same challenge, particularly since it can no longer expect the budget increases it had during the 1980s, it has yet to effectively achieve the same level of efficiency from its manufacturers. A major difference between DOD and commercial manufacturers is that DOD has, until recently, maintained its practice of inspecting rather than designing quality into a product while world-class companies have broadened their definition of quality to include design.
DOD is attempting to change its approach to quality by including design in the definition of quality. In translating this approach into practice, DOD will have to overcome a history in which many weapon system acquisitions have encountered significant cost and schedule overruns because of design and manufacturing problems. These problems usually resulted from acquisition strategies that began production before the design was complete and key manufacturing processes were in place and tested for capability. Some examples we have reported on follow: In 1985, we reported on four major weapon system acquisitions—the Army’s Copperhead projectile and Blackhawk helicopter, and the Navy’s High Speed Anti-Radiation Missile and Tomahawk missile—that had encountered substantial problems in early production, resulting in cost overruns and schedule delays. At least part of these cost and schedule overruns were caused by several untried manufacturing processes that were not studied to see if they could produce quality components. Symptoms of these problems were high scrap rates, parts shortages, and changes to engineering drawings.In 1988 and 1990, we reported that the B-2 bomber program’s immature design led to manufacturing problems in production that, in turn, led to schedule delays and cost increases. The contractors initiated thousands of changes to engineering drawings, some of which required new parts and tooling. Because the bomber was not ready for production, actual labor hours exceeded planned labor hours on the first three aircraft by 84, 86, and 94 percent, respectively, and the three major contractors estimated it would take over 10 million quality assurance labor hours to develop and produce 21 production aircraft, not including hours estimated for government oversight. These manufacturing problems delayed the first flight of the B-2 by 19 months and contributed to significant cost increases.In 1991, we reported that 12 tactical missile programs had cost and schedule overruns, partly because program offices had not adequately considered the risk associated with the weapons’ design, development, and production. One missile, the Advanced Medium Range Air-to-Air Missile, encountered problems in transitioning to production partly because certain electronic components proved too complex and had to be redesigned. This contributed to cost increases of 285 percent and a 5-year delay to the missile’s operational capability date.In 1994, we reported that quality problems with the C-17 program increased unit cost to production aircraft and delayed scheduled deliveries. Labor hours for rework and repair of production items made up 40 percent of the labor on the first five production C-17s, and scrap, rework, and repair cost about $44 million in 1993. In addition to these costs, some production aircraft were delivered to the Air Force with unfinished work and known deficiencies that had to be corrected after government acceptance. The Defense Science Board reviewed the C-17 program in 1993 and concluded that the production schedule could not be maintained unless the contractor changed its manufacturing and quality assurance processes. An Industry Review Panel on Specifications and Standards found that the C-17 production process had many quality problems that were adding cost to the program. It found that the production aircraft were being produced in a “development environment,” with unqualified processes, and with a reactive rather than proactive quality management system that did not analyze the causes of quality problems. It also found that a high number of engineering changes were making production less efficient. The review panel concluded that the program had an “inspection mentality” and that only 10 percent of quality cost was being spent on prevention.
These examples represent a persistent problem in DOD’s major acquisitions. All of the programs began producing weapon systems before their key characteristics were fully designed and the key processes for building the system were understood. Some started production before doing a significant amount of testing to determine if these systems would perform their required mission. The consequences of beginning production before completing testing have repeatedly included procurement of substantial inventories of unsatisfactory weapons that require costly modifications to achieve satisfactory performance and, in some cases, deployment of substandard systems to combat forces.
These examples do not necessarily condemn DOD to a repetition of the past or prejudge the potential success of DOD’s current efforts to better manage quality. However, they do underscore the challenge DOD faces in changing quality assurance practices. The systems in the examples were developed and produced using inspection-oriented quality assurance practices and significant DOD oversight. Yet, in each case, we reported that beginning manufacturing before the design was understood and manufacturing processes were controlled led to quality problems, cost increases, and schedule delays. The narrower interpretation of quality assurance that prevailed at the time may not have included the various design and engineering elements that played a part in the weapons’ eventual problems in production. However, the broader view of quality assurance practiced by leading commercial manufacturers includes these elements and thus covers a much larger range of responsibilities.
section_title: Companies Developed New Mindset: Prevent Problems Rather Than Fixing Them
paragraphs: Many companies have effectively evolved from an inspection-oriented quality assurance process, and the culture and infrastructure to support it, to a standard in which quality is an integral part of each stage of the design and production process.
The difference between how a manufacturer operates on a military contract versus how it operates on a commercial contract is often startling. For example, we visited a company that manufactured military and commercial products with similar specifications and uses; both manufacturing processes were in the same building. On one side of the hallway, the commercial process used automation and process control throughout the production process to continually reduce nonvalue-added inspections. On the other side of the hallway, the military process included two large test facilities at the end of the production process, used at least four test stations, and added about 10 days to the process. Company officials told us that the military process continued to have 100-percent end-item inspection, even though the quality of these products was not in question.
In response to increased competition in the 1980s, companies had to dramatically improve quality while reducing cost. They accomplished this by shifting paradigms. Rather than focusing on identifying and correcting problems, they began trying to focus on preventing the problems. Quality assurance changed from being a postmanufacturing step, done at the end of each process, to being part of the process itself. This was a significant culture change for these companies. Figure 1 illustrates the shift in paradigms from inspection-oriented to prevention-oriented quality practices.
Traditional quality assurance techniques relied upon many after-the-fact inspections, increasing costs in time and money. To remain profitable, manufacturers switched from detection to prevention-based quality strategies. These strategies teamed suppliers, manufacturing staff, and engineers to design quality into the product and to identify and control key characteristics. Prevention-based process control replaced end-item inspections.
To reduce their costs and gain an edge on the competition, companies have found that they have had to not only establish a basic commercial quality system, such as ISO-9000, as their baseline, but also consistently exceed it. Most of the companies we visited had obtained ISO-9000 certification, a basic quality management system that commercial customers are beginning to require. This standard ensures that a manufacturer has a well-documented commercial quality system. In addition, they all had developed or adopted advanced quality concepts that brought the concept of quality to the design phase of a product’s life. They began by using two advanced concepts called design for manufacturing and process control techniques. After a company had successfully incorporated each technique, it began to eliminate inspections—and the cost associated with them—and significantly reduced the amount of defects in its products. A third ingredient for success involved developing relationships with key suppliers, which ensured that parts and subcomponents were appropriate for a product’s design and arrived for production at consistently high quality and expected cost. Companies we visited had dramatic reductions in product defects—ranging from 34 to 90 percent—resulting from these techniques. Appendixes II and V contain detailed examples.
subsections: section_title: Design for Manufacturing Controls Product Development
paragraphs: Design for manufacturing represents a culture change that involves a whole new mindset. Rather than continuing the old practice of engineers designing a product in isolation and then handing it off to the manufacturing process, design for manufacturing involves all stakeholders in the process. The stakeholders form cross-functional teams that include representatives from the customer, marketing, research and development, engineering, manufacturing, key suppliers, quality assurance, finance, and customer support. Under the old practice, a product would require many design changes in full-scale production, creating additional defects, rework, and scrapped material. But now, the teams identify the requirements for a product’s performance and ensure that manufacturing processes are in place to meet those requirements within specified cost targets before production begins. Their objective is to build quality in up front rather than fix problems during some stage of the manufacturing process or discover there is no profitability. These cross-functional teams conduct phased development processes to ensure that a product’s design is producible and profitable. Continuous communication between the engineers who design the product and the people responsible for manufacturing it is a key to the process.
Using design for manufacturing techniques, these companies review projects to prevent a potentially unprofitable design from entering full-scale production. Teams use modeling and prototyping to determine the capability of existing manufacturing processes. As the development process continues, team members must make trade-offs between a product’s performance and its cost to meet strategic targets. For example, Texas Instruments’ semiconductor facility reviews a product’s potential profitability at each milestone of a five-phased development process. If any of these reviews indicate that cost targets cannot be met, the product’s development can be terminated. This eliminates additional investment in full-scale production tooling and facilities. Likewise, Varian Associates’ managers are given cost and schedule targets during the design phase, and special “out-of-bounds” reviews are held for behind-schedule or over-budgeted items throughout the development process. Figure 2 provides a conceptual model of the design for manufacturing process based on what we observed at these two companies.
section_title: Process Controls Help to Ensure Consistently High-Quality Products
paragraphs: Process control means controlling the production process by checking the quality while the work is being done. Beyond that, companies rely on final inspections of completed lots. Leading companies rely on “total process control,” which demands that every process is controlled by checking the quality during production. Rather than employing a lot of inspectors, however, they entrust the production workers to do it themselves.
The companies we visited had implemented similar systems. In addition to a basic quality system, such as ISO-9000, and an emphasis on including design as an element of the quality process, we found they (1) diffused responsibility for quality across the production line and (2) trained employees to use analytical diagnostic tools such as statistical process control, process mapping, or continuous flow manufacturing to maintain predictable processes.
The companies had developed or adopted advanced quality concepts that went beyond the basic ISO-9000 standard and that emphasized finding the cause of quality problems by gathering data and then eliminating those causes from the manufacturing process. Once root causes are discovered and processes are controlled, the likelihood of consistently high-quality products is significantly increased and the need for inspection at the end of the production line is reduced. For example, Cherry Electronics uses QS-9000, an advanced quality guide developed jointly by General Motors, Ford, and Chrysler as a required supplement to ISO-9000 for their suppliers. It includes comprehensive instructions for implementing design for manufacturing in new products and using process controls to reduce defects in final products. Cherry is undergoing certification for QS-9000 now and plans to require its own suppliers to become certified in QS-9000 as well. Also, Motorola and Varian Associates used the Malcolm Baldrige National Quality Award criteria to create advanced quality systems for themselves and their suppliers.
Representatives from both companies credit the use of these advanced quality guidelines for reducing defects while eliminating the need for end-item inspection. For example, Varian’s Nuclear Magnetic Resonance Instruments business unit heavily inspected both its product and its suppliers’ components coming into final assembly until the mid-1980s. At that time, it instituted process control and gradually reduced the number of its inspectors by 92 percent, from 26 to 2. Similarly, John Deere eliminated its quality assurance department by dividing the production line into “focused factories” and giving the inspection responsibility directly to each product manufacturing team.
section_title: Suppliers Become Part of the Team
paragraphs: Material from suppliers typically represents from 60 to 85 percent of the final product cost at the companies we visited. Because of this, companies focused on improving supplier quality so they could eventually reduce material defects and inspections. They began by reducing their number of suppliers. In addition, they created qualification and certification procedures that relied on periodic evaluations of supplier quality systems. Finally, they developed long-term relationships with valued suppliers, increasing communication and creating a partnership, when possible. These practices helped companies reduce suppliers’ defects by as much as 90 percent and inspections of incoming material by as much as 76 percent. Appendix III describes the results of our visits to specific companies that had implemented some of these supplier quality programs.
Commercial companies significantly reduced the number of suppliers—from 50 to 85 percent—to manage the cost and quality of incoming material more closely. Generally, a reduction in the supplier base eliminates poorer performers, increases the importance of top performers, and allows closer cooperation with suppliers in continuous improvement practices and in new product development. To help reduce the number of suppliers, companies generally have a qualification and certification process. The qualification process typically begins with an evaluation of each supplier’s current quality system and is often carried out by small cross-functional teams. For example, Varian’s Oncology Systems qualifies all new suppliers through an evaluation process that includes an assessment of suppliers’ defect rates, delivery performance, and a review of their internal quality systems.
Once a company determines that a supplier has processes in place to guarantee continued quality, it eliminates inspections and depends on periodic supplier quality information and reviews, generally not more than once annually. For example, Motorola uses its Quality System Reviews to assess the suppliers’ quality system every 2 years and score the supplier on such items as process controls and its ability to develop new products. The reviews are accomplished by a team of four or five quality management experts and take 2 to 3 days. The review guidelines use a scoring approach patterned after the Malcolm Baldrige National Quality Award criteria and include audit guidelines to determine how well a supplier controls quality in new product development and deploys good process controls. Both Motorola and Varian provide consulting teams and optional training courses to certified suppliers who either need or request assistance.
Most of the companies had strategies to enter into long-term relationships with world-class, high-quality suppliers, resulting in reduced inspection, planning, rework, and contracting costs. As suppliers progress in meeting cost and quality goals with companies, they are included on design teams, given schedule forecasts for future orders, and included in the company’s quality training program. In addition, longer term contracts typically help to reduce costs further and encourage valued suppliers to deliver material more efficiently. When companies introduce new products, valued suppliers are given the key characteristics of the part they are being asked to supply and then use this information to make sure their manufacturing processes are capable of making the component with consistently high quality.
section_title: DOD Moves to Improve Quality Assurance Practices
paragraphs: Beginning in the early 1990s, DCMC implemented Process Oriented Contract Administration Services (PROCAS) as a method of moving away from inspection-oriented quality assurance practices and toward process control. The intent of PROCAS is to team with contractors to identify, analyze, and manage production processes and reduce the need for oversight and inspection where it makes sense. In recognition that more advanced quality concepts are needed, a group of DOD officials and defense company representatives have joined in an initiative that, if implemented, could eliminate the costs of redundant quality assurance processes. According to the plan created by the Government and Industry Quality Liaison Panel in April 1995, DOD would allow a contractor to use the same quality management system—based on process controls—for its military contracts that it uses on its commercial contracts. Related to the panel’s goals, DOD’s Single Process Initiative, introduced in December 1995, also supports moving from multiple government-unique management systems to a single management system common to both commercial and military contracts. In addition, DOD has established a policy for using integrated product and process development concepts to integrate all acquisition activities, from product concept through production and field support. How well it will support the panel’s goals remains to be seen.
The Liaison Panel’s plan has three overall goals. The first goal is to develop a single quality system that can meet commercial or military quality requirements. It envisions a multitier quality management framework based on criteria similar to the commercial ISO-9000 standard that would be recognized by all government and commercial entities. It begins with a basic quality system that must use process controls from design to delivery and would be reviewed on a regular basis by a single, unified government audit team. As a contractor moves to advanced quality concepts, such as design for manufacturing, and commits to continuous improvement of all its processes, its advanced quality system would be reviewed by government and industry representatives. One objective of this goal is to create a culture that can ensure effective design, production, and delivery processes without constraining suppliers with a set of inspection and oversight requirements.
The plan’s second goal is to have government and industry share the most advanced quality concepts in defining requirements for, designing, and manufacturing military products. It envisions a system that will encourage continuous improvement across the industry by identifying the most advanced methods of ensuring quality, making these concepts available to all contractors, and helping train personnel in these concepts. DOD and industry agreed that contractors who can provide evidence of using these advanced concepts to ensure quality should be given credit during source selection.
The plan’s third goal is to establish and implement efficient oversight methods. It envisions the government and industry developing a single set of criteria to evaluate contractors’ quality management systems. This criteria should be implemented in a unified evaluation process and should promote effective and efficient innovation. Most importantly, DOD believes the criteria and its implementation method should be accepted by all government customers to avoid inconsistency and duplication of quality evaluations. The Co-chair of the Government and Industry Liaison Panel stated that this evaluation criteria would be used by government audit teams, perhaps from DCMC, in registering contractors’ basic and advanced quality systems rather than current methods of oversight. Appendix IV lists the 15 specific tasks that the panel has undertaken to achieve its goals, their current status, and their projected completion dates.
In December 1995, DOD began the Single Process Initiative, managed by DCMC, that allows contractors with military contracts to transition their quality system from MIL-Q 9858A to their best practice, such as a quality system based on ISO-9000, the basic commercial standard. The response to date has been slow; as of June 5, 1996, 38 contractors had submitted proposals to change their quality systems, 5 of which had been approved by DCMC. In discussions with government officials, we found that the biggest reason for this slow response is cultural. The defense community’s traditional quality assurance practices have been inspection-based; the newer, more advanced concepts advocate process control. In addition, some contractor officials believed there is an understandable fear among DOD employees that changes in the quality assurance strategy will translate to a loss of jobs. Also, defense contractors do not see the benefit of implementing quality systems that may result in payment to the government for savings resulting from more efficient practices.
DOD’s approval of five contractors’ commercial quality systems means that they are accepted as the basic quality system on all government contracts. This is a positive step toward introducing more advanced quality concepts from the commercial world that force quality considerations during the design phase of an acquisition. However, in agreeing to this change, DCMC reserved the right to review all quality documentation at any time; perform any inspections, verifications, or evaluations it deems necessary; review any supplies from other facilities; and disapprove the quality system or any portion of it. The method or frequency with which DCMC invokes these rights will be as important as a change in the military standard. Also, it is important that DOD continue to move contractors beyond changing basic quality systems, toward advanced concepts.
According to DCMC officials, DCMC approved changes to commercial quality systems on 4,255 out of 158,000 existing military contracts—less than 3 percent. It approved ISO-9000 on 398 new military contracts between May 1995 and April 1996. DCMC approves an average of 36,000 contracts per year, meaning that about 1 percent of new contracts in the past year have been approved with ISO-9000 as the quality standard.
section_title: Recommendations
paragraphs: The commercial world has proven that it can design and manufacture consistently high-quality products by focusing on building quality into the product’s design, understanding the key characteristics of the product and the manufacturing processes necessary to build it, training production personnel to control those processes throughout production, and instituting quality programs with suppliers based on these same principles. We saw convincing evidence that these practices improve product quality and reduce time and labor spent on quality assurance oversight by making it unnecessary.
DOD recognizes the benefits of taking a broader approach to quality assurance. It has made some policy changes and is beginning to implement new practices. If DOD provides incentives for implementation of the advanced commercial practices, such as those identified in this report, we believe it can significantly improve quality, reduce costs of its acquisition programs, and apply savings to future modernization efforts. However, we do not believe DOD’s quality assurance culture will change easily. Our conclusion is based on discussions with DOD and service representatives and our review of past acquisitions where DOD and we repeatedly identified unstable design, poor process controls, and poor transition to production as causes for manufacturing quality problems and made recommendations that have not been implemented.
Therefore, we recommend that the Secretary of Defense (1) establish measurable steps to implement and monitor the progress of the Government and Industry Quality Liaison Panel plan closely; (2) periodically assess its success in implementing basic standards such as ISO-9000; (3) develop ways to encourage the adoption of advanced quality concepts of design for manufacturing, process controls, and supplier quality programs throughout the defense industry, using commercial practices as a guide; and (4) as suggested in the plan, establish incentives for defense contractors to participate, such as providing credit during source selection for successful implementation of these advanced quality practices. Such selection criteria could also provide world-class companies greater opportunities to participate in DOD’s weapons acquisition programs.
To assist DOD in changing its own quality assurance culture, we recommend that the Secretary of Defense expeditiously determine who in DOD’s acquisition community can best oversee the advanced quality functions used by defense contractors in developing and producing weapon systems, using commercial practices as a guide in assigning these functions, and provide all necessary training for any new responsibilities that DOD personnel need to perform.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD agreed with the intent of the report and stated that it has already implemented many changes in line with our recommendations. (See  app. VI.) These include the institution of integrated product and process teams in revised acquisition policy directives and the development of metrics to measure improvement in the overall acquisition process. DOD stated that it believed that no additional actions were required in response to our recommendations at this time.
We agree that these and other actions initiated by DOD represent positive steps in reforming the quality assurance process and are consistent with the intent of our recommendations. However, time is needed to determine whether these steps translate into tangible changes in quality assurance practices on individual programs. As we note in the report, a number of major acquisition programs over time have failed to include quality considerations in the design phase. Although the prevailing standard—MIL-Q-9858A—for these programs allowed some latitude for interpreting how quality assurance could be carried out, it was the actual practice—not the guidance—that was more narrowly focused on inspections. These experiences underscore the challenge DOD faces in implementing advanced quality concepts.
section_title: Scope and Methodology
paragraphs: To develop information for this report, we interviewed and obtained documents from officials of the Office of the Secretary of Defense in the Pentagon and DCMC at Fort Belvoir, Virginia, because of the quality assurance policymaking responsibilities and initiatives that are ongoing at that level. We also held several discussions about quality assurance at the service level. We discussed commercial quality assurance practices with officials from the following commercial manufacturing organizations:
Delco Electronics, Milwaukee, Wisconsin;
John Deere Horicon Works, Horicon, Wisconsin;
Cherry Electrical Products, Waukegan, Illinois;
Texas Instruments Lubbock Metal-Oxide Semiconductor, Lubbock, Texas;
Varian Nuclear Magnetic Resonance Instruments, Palo Alto, California;
Varian Chromatography Systems, Walnut Creek, California;
Varian Oncology Systems, Palo Alto, California; and
Motorola Paging Products, Boynton Beach, Florida.
A detailed description of the companies we visited is contained in appendix I.
We then developed a data collection instrument that would assist us in gathering uniform, quantifiable measurements about the techniques these organizations used to improve operations and the results they accomplished. We visited these manufacturing organizations, followed the same agenda with each, and gathered the same data at each organization.
We also visited Texas Instruments Defense Systems & Electronics, Dallas, Texas, a defense contractor. In addition, we visited DELCO in Milwaukee, Wisconsin, to discuss differences between military and commercial practices in a broader context.
We reviewed literature and various databases provided by the American Productivity Quality Center’s International Benchmarking Clearinghouse to identify manufacturing organizations that have shown significant improvement in quality while reducing oversight functions such as supplier oversight and end-item inspections. Our discussions centered around their overall quality plan and the techniques they used to ensure supplier quality, producibility of new products, and control of their final assembly processes while reducing nonvalue-added cost.
We performed our review from August 1995 to July 1996 in accordance with generally accepted government auditing standards.
We are sending copies of this report to congressional committees; the Secretaries of the Army, the Navy, and the Air Force; the Director, Defense Logistics Agency; and the Director, Office of Management and Budget. We will also make copies available to others upon request.
Please contact me at (202) 512-4383 if you or your staff have any questions concerning this report. The major contributors to this report were Michael J. Sullivan, Shari A. Kolnicki, Gordon W. Lusby, and Carolyn S. Blocker.
section_title: Companies We Visited as Part of the Review
subsections: section_title: Delco Electronics
paragraphs: Delco Electronics, Milwaukee, Wisconsin, manufactures products for both commercial and military customers. Delco produces avionics circuit boards and assemblies for several aircraft and light armored vehicles. For its commercial automotive customers, it produces power train, fuel-injection, and remote function computers.
section_title: John Deere Horicon Works
paragraphs: John Deere Horicon Works, Horicon, Wisconsin, has worldwide responsibility for engineering lawn and grounds care products.
section_title: Cherry Electrical Products
paragraphs: Cherry Electrical Products, Waukegan, Illinois, designs and manufactures a broad range of electrical and electronic components, assemblies, and systems for the automotive industry. Applications include interior use, weather-exposed devices, and under-the-hood functions. Cherry Electrical received the General Motors’ Targets for Excellence Award and two Q1 Preferred Quality Awards from Ford Motor Company’s Diversified Operations. Cherry Electrical has applied for the Malcolm Baldrige award twice. The overall sales of the Cherry Corporation in 1994 were over $339 million.
section_title: Texas Instruments Lubbock Metal-Oxide Semiconductor
paragraphs: Texas Instruments Lubbock Metal-Oxide Semiconductor, Lubbock, Texas, designs and manufactures integrated circuits that are packaged and assembled overseas. Electronic Programmable Read-Only Memory, a chip that can be used in computers, telephones, automobiles, and other computerized systems, is Texas Instruments’ best selling item. In 1993, the overall sales for Texas Instruments Corporation were $8.5 billion.
section_title: Varian Chromatography Systems
paragraphs: Varian Chromatography Systems, Walnut Creek, California, supplies gas and liquid chromatography, gas chromatography, and mass spectrometry instruments as high-precision, high-end analytical tools. Its customers are pharmaceutical, chemical, environmental, and other research and developmental laboratories and quality laboratories worldwide. The company manufactures 11 different product lines, with a total volume ranging from 200 to 1,500 units sold per year. The corporation’s total sales in 1995 were over $1.5 billion.
section_title: Varian Nuclear Magnetic Resonance Instruments
paragraphs: Varian Nuclear Magnetic Resonance Instruments (NMRI), Palo Alto, California, manufactures scientific and medical instruments that determine the composition of substances through atomical analysis using nuclear magnetic resonance. Using such instruments, scientists determine the atomic connectivity, atomic spatial orientation, and dynamic characteristics of molecules. The analysis is useful for chemical, biomolecular, material science, and biomedical fields. NMRI’s major markets consist of three customer bases: academia; private industry, including pharmaceutical, basic chemistry, biotechnology, and petrochemical industries; and the federal government. NMRI manufactures low-volume, high-precision products and sells approximately 250 units each year. Prices range from $1,000 to $3 million. It was included in Varian’s corporate application for the Malcolm Baldrige National Quality Award in 1992, 1993, and 1994.
section_title: Varian Oncology Systems
paragraphs: Varian Oncology Systems, Palo Alto, California, develops and manufactures linear accelerators as its principal product. Linear accelerators are high-technology instruments that produce X-rays, electrons, and other high-energy particles used to treat cancer. Approximately 30 percent of its remaining business is in computerized treatment planning and information systems. Principal customers include universities, hospitals, government institutions, and free-standing clinics. Varian Oncology’s annual sales are approximately $350 million. Varian Oncology Systems applied for the Malcolm Baldrige Award in 1995. It has received other accolades, such as mention in the “Top Ten” plants in Industry Week in 1993, the California Governor’s Golden State Award for Quality in the Marketplace in 1994, the Best Factory Award from Management Today in 1992, and the European Quality Award Commendation in 1993.
section_title: Motorola Paging Products Group
paragraphs: Motorola’s paging plant in Boynton Beach, Florida, is part of Motorola’s Messaging Information and Media Sector business unit. This business unit designs, manufactures, and distributes a variety of messaging products, including pagers and paging systems, wireless and wireline data communication products, handwriting recognition software, and infrastructure equipment, systems, and services. The plant in Boynton Beach handles the strategic and tactical dealings with suppliers. Motorola developed a successful six-sigma manufacturing process and was the first Malcolm Baldrige award winner in 1988. Motorola’s resulting successes have inspired many other U.S. corporations to use it as a quality benchmark. As a corporation, Motorola’s net sales in 1995 were $27 billion.
section_title: Description of Companies’ Process Control Practices in Design and Manufacturing and Their Benefits
paragraphs: Texas Instruments Lubbock Metal-Oxide Semiconductor - Lubbock, Texas The following processes are used to ensure a product’s profitability and producibility before it enters production: — Profit and quality team is involved throughout the process. — Manufacturing and design departments work together.  — Manufacturing staff determine critical parameters to achieve consistent quality while engineers use statistical process control to     discover whether a parameter remains in control if a change in the process or product occurred. — Engineers consult with manufacturing staff during the design phase in an effort to prevent critical failures from occurring in     production.
Process controls have helped Texas Instruments  — eliminate end-item inspection that no longer adds value and reduce inspectors from 16 to 12, — reduce manufacturer’s defects by over 68 percent, and — increase productivity by over 120 percent.
Varian Nuclear Magnetic Resonance Instruments - Palo Alto, California Varian’s NMRI business unit’s approach is as follows: — Baldrige Award criteria and ISO-9000 standards are used as the foundation of its quality system. — Cross-functional teams integrate design, manufacturing, and field service issues during product development.  — Statistical process control is used to develop statistically significant boundaries within which the product remains functional. — Software has been instrumental in improving the productivity and accuracy of the testing process. — NMRI has sought to eliminate inspections by relying on self-testing and continuous improvement.
Process control initiatives have allowed NMRI to — eliminate its quality assurance department, — empower on-line operators with in-process control, — decrease the number of inspectors from 26 to 2, and — increase productivity by 97 percent over 6 years.
Texas Instruments Defense Systems and Electronics - Dallas, Texas Texas Instruments Defense Systems and Electronics won the Malcolm Baldrige Award in 1992 with the following approach: — Design for manufacturing was used in defense programs to reduce part counts, increase opportunities for automation, and simplify     assembly. — Multifunctional teams formed during the concept phase of a product’s life cycle remain together throughout the production     program.  — Production process is controlled by statistical process control, which identifies the critical parameters. — Continuous flow manufacturing is used to identify bottlenecks and nonvalue-added costs in production lines.
Since 1991, Defense Systems and Electronics has — reduced defects by almost 70 percent, — reduced scrap rates by 50 percent, — reduced inspectors by 50 percent, and — increased productivity by about 30 percent. (continued)
Varian Oncology Systems - Palo Alto, California Varian’s Oncology Systems business unit used the Malcolm Baldrige Award criteria as a foundation for its quality strategy.  — Five-phased product development process uses cross-functional teams. — Program reviews check progress and readiness, and senior management reviews the project every 6 weeks.  — Capability of key manufacturing processes is measured before production begins. — All production processes are mapped to determine nonvalue-added steps.
Using these techniques, Varian was able to — reduce final test hours of each high-energy linear accelerator from 700 to 200; — increase the maximum production of these items from 110 to 200; — reduce significantly, the defects at the end of assembly; and — reduce inspectors by 94 percent.
Cherry Electrical Products - Waukegan, Illinois Cherry Electrical Products experienced decreased labor costs and increased quality in its stamping department using statistical process control. For example: — Using statistical data, the company diagnosed a problem with the start-up of the machine, compensated for it, and reduced the     number of inspectors monitoring the process from four to two. Furthermore, roving inspectors in the department and final     inspection on the end item were eliminated, and scrap and waste were significantly reduced. — Cherry reduced the number of operators from 24 to 1 on its spring generator machines while increasing quality. Diagnostic,     analytical problem-solving using statistical process control showed the company’s defects and scrap problems were not a     result of machine variability as previously thought, but due to variability introduced by numerous operators. Because of     statistical process control, labor was reduced, quality was increased, and waste was cut.  — Because of product consistency resulting from process controls, one department was able to reduce its headcount from 20 to 12.
Overall, Cherry, by using statistical process control, has brought its quality processes under control and decreased inspectors by  78 percent, while reducing defects by 90 percent.
Varian Chromatography Systems - Walnut Creek, California Varian Chromatography used “Andon,” a Japanese quality process control, to improve its processes. Using Andon, each operator is empowered to signal a problem to the rest of the manufacturing team. Each station is equipped with red, yellow, and green lights, with a red light empowering the employee to stop production. — Operators record any problems found on the production line on an opportunity board.  — Weekly meetings are held with operators to resolve problems in cross-functional teams. — Varian cross trains its employees to increase their skill base so that they are able to understand the entire process.
Varian reduced waste and increased savings through these process control techniques. For example, it — reduced quality test time by 98 percent, from a 96-hour test to a 2-hour test; — reduced defects by 34 percent;  — decreased the number of inspectors by over 75 percent; and — improved productivity by 39 percent since 1992.
Varian Chromatography Systems - Walnut Creek, California Varian Chromatography Systems’ supplier quality practices are to — concentrate its business with a few, best-qualified suppliers; — conduct just-in-time manufacturing with them, as part of its Value Managed Relationships program; — perform zero-receiving inspection and not stock the suppliers’ parts; instead, components from suppliers are coordinated daily,     and the supplier’s pretested, unboxed parts are delivered directly from the truck to Varian’s manufacturing assembly line; — provide suppliers a total year goal and give a rolling 12-month forecast instead of the company preordering its material in a     traditional “push” system;  — give copies of its master production schedule to suppliers; and  — implement a certification process whereby all new components must pass first article inspection and three defect-free lot inspections     prior to approval. In addition, approved suppliers are subject to annual review and quarterly feedback reports.
Through its supplier quality program, — inventory has been reduced by 68 percent; — supplier defects have decreased 75 percent; — suppliers were reduced by 78 percent, down from over 2,000 to approximately 440; and — suppliers are kept for “life” as long as they provide the technological capabilities, and some have been supplying the company for     as long as 30 years, with many over 10 years.
Varian Oncology Systems - Palo Alto, California Varian Oncology Systems’ supplier quality practices are to — deliver 100-percent quality parts directly to the factory floor; — eliminate receiving inspections for certified suppliers and conduct periodic on-site reviews; — assist any certified supplier experiencing quality or delivery problems with supplier corrective action teams; — provide annual performance reports, periodic report cards, and performance awards as feedback to its suppliers; — require certified suppliers to maintain competitive prices, high quality levels, and excellent delivery performance to retain     certification; and — provide certified suppliers with long-term contracts, access to Varian training in process controls, and improved schedule visibility.
Oncology Systems conducts business with 45 certified suppliers and estimates that these relationships have saved $3.3 million over the past 3 years due to eliminated inspections, reduced planning and purchasing requirements, and reduced defects.  It regularly invites suppliers to join continuous process improvement teams that are focused on process control techniques. Since 1990, supplier defects have been reduced by 73 percent, and receiving inspections have been significantly reduced.
Varian Nuclear Magnetic Resonance Instruments - Palo Alto, California Varian Nuclear Magnetic Resonance Instruments has increased both savings and supplier quality by — switching from an inspection-oriented environment to one of process control; — training its workers in statistical process control factorywide and using it to internally track the performance of its suppliers; — reducing its supplier base by over 60 percent by weeding out those with poor performance records and those it seldom used; and — conducting certification programs where suppliers are required to pass five to six lot inspections. Varian-approved suppliers’parts     completely bypass receiving inspection, but if they do not meet quality standards as a product is assembled on the production     line, they are immediately rejected.
Due to the supplier program, supplier defect rates have decreased by 96 percent, and inspections have decreased by 76 percent. In fact, the company is so confident in the supplier’s quality for one critical component that comprises 80 percent of the total cost of supplier material, that Varian does not perform final assembly and test until after the product is delivered to the customer. Thus, Varian and its customer view the results of final assembly and test simultaneously. (continued)
Motorola Paging Products Group - Boynton Beach, Florida Motorola’s Paging Products Group supplier quality practices are to — conduct certification programs requiring that each supplier undergo an on-site evaluation and submit sample parts for approval, — conduct a quality system review at the end of the first year to determine the supplier’s status, — eliminate receiving inspections after suppliers have passed inspection defect-free on three lots of material, — require preferred suppliers to apply for the Baldrige Award, — emphasize long-term relationships with suppliers, — train all of its suppliers in total quality techniques, — share its schedule forecasts with suppliers, and — conduct early supplier involvement programs on new products.
As a result of its supplier quality assurance program, Motorola has — eliminated all inspection of incoming material from all but new suppliers;  — reduced suppliers by 85 percent, from 800 to 118; and — decreased supplier defects by over 90 percent over a 7-year period.
Texas Instruments Lubbock Metal-Oxide Semiconductor - Lubbock, Texas Texas Instruments Metal-Oxide Semiconductor ensures the quality of supplier products by — relying upon third-party certification to periodically review its suppliers; — using a Sematech database, allowing TI to choose from among the best suppliers; Sematech is a semiconductor consortium of      10 large companies that periodically conducts quality system audits of suppliers and retains past performance information     so that its members can increase their effectiveness in selecting and retaining the few best ones; and  — depending upon the Texas Instruments purchasing center located in Dallas, which screens and approves suppliers and parts as it     does for all wafer fabrication facilities.
Because of this supplier process control program, Texas Instruments has experienced virtually zero defects from its suppliers. It is also able to bypass inspecting parts from high quality suppliers whose processes are in control. These parts are directly used by Texas Instruments-Lubbock in its products.
Cherry Electrical Products - Waukegan, Illinois Cherry Electronics uses the following supplier quality practices: — Most of Cherry’s 400 suppliers are “approved.” — The approved suppliers follow Cherry’s Supplier Quality Assurance Manual. — New suppliers and/or new items are audited, via a layout inspection, and statistical process control analysis is performed. — Cherry matches its findings against those of the suppliers’ and if a match results, the supplier’s item becomes “approved.” — The relationship with approved suppliers is based on trust and self-monitoring by the suppliers, with statistical process control     data available from suppliers upon request.
As a result of its supplier quality practices, Cherry has  — reduced the number of suppliers by 50 percent, from 800 to 400 suppliers; — reduced the number of defects from suppliers by 90 percent; and — reexamined and eliminated unnecessary specifications.
John Deere Horicon Works - Horicon, Wisconsin John Deere’s Horicon Works supplier practices are the following: — Suppliers of critical parts must have established +/-3 sigma quality. (A sigma unit is a measure of scale that can be used for     quality data. Three sigma quality (+/-3) is equal to three standard deviations away from an average, and the area covered     by three sigma is 99 to 100 percent of the data. Under three-sigma quality, a supplier product’s values will lie within a     known, predictable range 99 to 100 percent of the time.)  — Deere measures the parts for conformance and then requires process variability reports for the key characteristics of the part from     the supplier on a periodic basis. — Deere looks for strategic partners for its critical parts who will share management philosophies, be industry leaders, and require     very little communication or reviews.
As a result, Deere has reduced its supplier base 81 percent, from over 800 to 151, and has qualified 40 of them. It has been able to eliminate receiving inspections as a result, a savings of 25 positions.
To be signed.
Signed on April 24, 1995.
Develop pilot program characteristics and candidates.
To be implemented  November 1995.
No status update.
Identify changes required to FAR parts 46 and 52 as well as DOD FAR Supplement (DFARS) part 246.
DFARS changes - December 1995 FAR changes - July 1995.
DFARS case 246 approved; inputs needed to FAR part 46. -Identify key elements of a quality plan. -Write data item description incorporating needed elements. -Develop revisions needed to ANSI/ASQC Q9000.
Obtain data item description approval.
Data item description 81449 approved January 1995.
Develop government/industry training template for quality system.
Plan of action for training - May 1995.
Completed draft - May 1996.
Develop criteria for government and industry to evaluate contractor’s basic quality management system.
Complete criteria August 1995. Completed draft - May 1996.
7. Develop evaluator guidelines. Determine if common evaluation criteria exists among audit community.
Complete guidelines March 1995.
Completed draft - May 1996.
Determine the most efficient way of performing the oversight function to meet the criteria for evaluating quality management systems.
Develop guidelines for oversight - September 1995.
Completed draft - May 1996.
Completed draft - May 1996.
Develop a handbook that contains criteria or guidelines that can be used to encourage the use of advanced quality concepts.
Distribute handbook for concurrence - September 1995.
Recognize, improve, and promote the use of value-adding advanced quality practices.
Final report - September 1995. Completed draft - May 1996.
Establish an advanced quality practices clearinghouse to help promote quality management systems in government and industry.
Implement clearinghouse and publicize - October 1995.
Final report prepared February 1996. (continued)
Draft governmentwide agreement to mutually recognize a contractor’s quality system based on defined baseline requirements.
Memorandum ready October 1995.
Final draft contingent upon completion of all other tasks.
Establish integrated government audit system whereby audit results are shared by all participating government agencies.
Not specified.
Ongoing.
Courses to start March 1996.
Ongoing.
Develop four courses based on the approved ISO 9000 training template.
Obtain review and approval of course content and context.
Review to start December 1995. Ongoing.
section_title: Examples of Savings Using Statistical Process Control
subsections: section_title: Description of Process Steps
paragraphs: process were not critical to the product’s quality. The elimination of these six measurements saved a total of 36 labor hours for each product manufactured. Because of this application of statistical process control, rework decreased from 3 to 5 percent to 0.5 percent, the plant experienced a 50-percent decrease in the product’s variability, and a 50-percent decrease in cycle time. Because of this decreased cycle time, the resulting throughput was increased, the plant became more efficient, and Texas Instruments avoided purchasing an extra scanning electron microscope costing $800,000, as well as saved the cost of a technician that would have been needed to operate it.
section_title: Related GAO Products
paragraphs: Weapons Acquisition: Low-Rate Initial Production Used to Buy Weapon Systems Prematurely (GAO/NSIAD-95-18, Nov. 21, 1994).
Military Airlift: The C-17 Program Update and Proposed Settlement (GAO/T-NSIAD-94-166, Apr. 19, 1994).
Weapons Acquisition: A Rare Opportunity for Lasting Change (GAO/NSIAD-93-15, Dec. 1992).
Tactical Missile Acquisitions: Understated Technical Risks Leading to Cost and Schedule Overruns (GAO/NSIAD-91-280, Sept. 17, 1991).
Why Some Weapon Systems Encounter Production Problems While Others Do Not: Six Case Studies (GAO/NSIAD-85-34, May 24, 1985).
Strategic Bombers: B-2 Program Status and Current Issues (GAO/NSIAD-90-120, Feb. 22, 1990).
Effectiveness of U.S. Forces Can Be Increased Through Improved Weapon System Design (GAO/PSAD-81-17, Jan. 29, 1981).
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.