Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality
Published: 2008-02-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: A Senate report related to the National Defense Authorization Act for Fiscal Year 2007 asked GAO to compare quality management practices used by the Department of Defense (DOD) and its contractors to those used by leading commercial companies and make suggestions for improvement. To do this, GAO (1) determined the impact of quality problems on selected weapon systems and prime contractor practices that contributed to the problems; (2) identified commercial practices that can be used to improve DOD weapon systems; (3) identified problems that DOD must overcome; and (4) identified recent DOD initiatives that could improve quality. GAO examined 11 DOD weapon systems with known quality problems and met with quality officials from DOD, defense prime contractors, and five leading commercial companies that produce complex products and/or are recognized for quality products.
section_title: What GAO Found
paragraphs: Problems related to quality have resulted in major impacts to the 11 DOD weapon systems GAO reviewed--billions in cost overruns, years-long delays, and decreased capabilities for the warfighter. For example, quality problems with the Expeditionary Fighting Vehicle program were so significant that DOD extended development 4 years at a cost of $750 million. The F-22A fighter aircraft experienced cracks in the plane's canopy that grounded the flight test aircraft, and initial operating capability for the Wideband Global SATCOM satellite was delayed 18 months because a supplier installed some fasteners incorrectly. GAO's analysis of 11 DOD weapon systems illustrates that defense contractors' poor practices for systems engineering activities as well as manufacturing and supplier quality problems contributed to these outcomes. Reliance on immature designs, inadequate testing, defective parts, and inadequate manufacturing controls are some of the quality problems that GAO found. Senior prime contractor officials GAO met with generally agreed with GAO's assessment of the causes of the quality problems. In contrast, leading commercial companies GAO contacted use more disciplined systems engineering, manufacturing, and supplier quality practices. For example, rather than wait to discover defects after the fact, Boeing Commercial Airplanes tries to design parts that can be assembled only one way. Effective use of many systems engineering practices has helped Space Systems/Loral, a satellite producer, improve overall quality, for example, by allowing the company to operate its satellites for more than 80 million consecutive hours in orbit with just one failure. Companies also put significant effort into validating product design and production processes to catch problems early on, when problems are less costly to fix. They conduct regular audits of their suppliers and hold them accountable for quality problems. DOD faces its own set of challenges--setting achievable requirements for systems development and providing effective oversight during the development process. In conducting systems development, DOD generally pays the allowable costs incurred for the contractor's best efforts. These conditions contribute to an acquisition environment that is not conducive for incentivizing contractors to build high-quality weapon systems and DOD, which typically uses cost-reimbursement contracts to develop weapon systems, assumes most of the risks and pays contractors to fix most of the problems. DOD has taken steps to improve its acquisition practices by experimenting with a new concept decision review practice, selecting different acquisition approaches according to expected fielding times, and establishing panels to review weapon system configuration changes that could adversely affect program cost and schedule. None of these initiatives focus exclusively on quality issues, and none specifically address problems with defense contractors' practices.

section_title: Letter
paragraphs: The Department of Defense (DOD) plans to invest about $1.5 trillion (in  2007 dollars) in its current portfolio of major weapon systems. However,  the cost of designing and developing these systems could continue to  exceed estimates by billions of dollars if DOD continues to employ the  same acquisition practices, including those for quality, as it has in the past.  Excessive scrap, rework, and repair costs, as well as reliability problems  impact overall quality and could ultimately present serious consequences  on a weapon system’s long-term support costs and affordability.
Like DOD, commercial companies collectively invest trillions to develop  their products. Fundamentally, they know they must meet customer  expectations to ensure continued growth. Many leading companies follow  a knowledge-based approach for product development and rely on proven  practices to attain high-quality products, control costs, and make a profit.  While commercial companies are not without flaws and can produce poor- quality products, the demands of the marketplace force them to place a  high priority on quality.
This report examines how DOD and its defense contractors can improve  the quality of major weapon systems. A Senate report related to the John  Warner National Defense Authorization Act for Fiscal Year 2007 asked  GAO to compare quality management practices used by DOD and its  contractors to those used by leading commercial companies and make  suggestions as to how DOD’s practices could be improved. We (1)  determined the impact of quality problems on selected DOD weapon  systems and defense contractors’ practices that contributed to the  problems; (2) identified practices used by leading commercial companies  that can be used to improve the quality of DOD weapon systems; (3)  identified problems DOD faces in terms of improving quality; and (4)  identified recent DOD initiatives that could improve quality.
To do this, we considered quality activities that take place from the time  requirements are established for a product until it is fielded.  This includes  virtually all key design and engineering elements during development, the  transition to production, and production itself. We examined 11 DOD  weapon systems with known deficiencies that are in later phases of  development or production to determine the impact of quality problems  and assessed the effectiveness of defense contractors’ quality practices for  these systems. The 11 weapon systems were chosen to demonstrate the  types of problems DOD weapon systems experience and to help focus our  discussions with leading commercial companies on aspects of  development that caused DOD major quality problems.  We also met with  representatives from the Office of the Under Secretary of Defense  (Acquisition, Technology and Logistics), each of DOD’s military services,  selected commands, the Defense Contract Management Agency, and six of  DOD’s largest prime contractors—BAE Systems, Boeing Integrated  Defense Systems, General Dynamics, Lockheed Martin, Northrop  Grumman, and Sikorsky Aircraft—to discuss quality practices used to  build DOD weapon systems and to obtain related documentation. These  prime contractors are involved in a little over $1.1 trillion, or about 76  percent, of DOD’s expected $1.5 trillion expenditure on weapon systems in  its current portfolio. To identify leading commercial companies’ quality  practices, we interviewed and obtained documentation from quality  management personnel at five companies:  Boeing Commercial Airplanes;  Cummins Inc., a manufacturer of diesel and natural gas-powered engines;  Siemens Medical Solutions, a producer of ultrasound systems; Space  Systems/Loral, a producer of satellite systems; and Kenworth, a trucking  company. Much of the information we obtained from these companies is  anecdotal, due to the proprietary nature of the data that could affect their  competitive standing. We also met with officials from American Airlines  and Intelsat, a satellite communications company, to understand the  commercial customers’ role in acquiring high-quality products.
We compared leading commercial company practices with those used by  DOD and the prime contractors we reviewed to identify both potential  areas for improvement and practices that could improve the quality of  DOD weapon systems. We also reviewed recent DOD initiatives aimed at  improving acquisitions to determine if they have the potential to improve  weapon system quality. Appendix I includes additional details about our  scope and methodology. We conducted this performance audit from  September 2006 to December 2007 in accordance with generally accepted  government auditing standards.  Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives.  We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In general, a quality product is one that is delivered on time, performs as  expected, and can be depended on to perform when needed, at an  affordable cost. This applies whether the customer is an individual  purchasing a simple consumer good, such as a television, a hospital  purchasing medical imaging equipment to help doctors treat cancer  patients, or DOD purchasing sophisticated weapons for its warfighters to  use on the battlefield.
For about 3 decades, DOD based its quality requirements on a military  standard known as MIL-Q-9858A, and its quality assurance practices were  oriented toward discovering defects through inspections. In 1994, the  Secretary of Defense announced that commercial quality standards should  replace MIL-Q-9858A. The intent was to remove military-unique  requirements that could present barriers to DOD in accessing the  commercial supplier base. Currently, responsibilities for quality policy and  oversight fall under the Systems and Software Engineering organization,  within the Office of the Secretary of Defense.
Over the past 20 years, commercial companies have had to dramatically  improve quality in response to increased competition. Many companies  moved from inspection-oriented quality management practices—where  problems are identified and corrected after a product is produced—to a  process in which quality is designed into a product and manufacturing  processes are brought in statistical control to reduce defects. Many  companies have also adopted commercial quality standards, such as ISO  9001. This standard was developed by the International Organization for  Standardization, a non-governmental organization established in 1947 to  facilitate the international coordination and unification of industrial  standards. Similar to DOD’s MIL-Q-9858A, ISO 9001 includes requirements  for controlling a product’s design and development, and production, as  well as processes for oversight and improvement. Some industries, such as  the automotive and aerospace industries, also have standards specific to  their sector based on the ISO 9001. Because supplier parts account for a  substantial amount of the material value of many companies’ products,  companies may require their suppliers to adopt the same standards.
In practice, DOD and its prime contractors both participate in activities  that contribute to weapon system quality. DOD plays a large role in quality  when it sets key performance parameters, which are the most important  requirements DOD wants prime contractors to focus on during  development. For example, if reliability is one of those key performance  parameters, then prime contractors are expected to focus on it during  weapon system design. Prime contractors employ quality assurance  specialists and engineers to assess the quality and reliability of parts they  receive from suppliers, as well as the overall weapon system. DOD has its  own quality specialists within the Defense Contract Management Agency  and the military services, such as the Navy’s Supervisor of Shipbuilding  organization. DOD’s quality specialists oversee prime contractors’ design,  manufacturing, and supplier management activities; oversee selected  supplier manufacturing activities; and conduct final product inspections  prior to acceptance.
GAO previously reported on DOD quality practices in 1996. At that time,  we reported that numerous weapon system programs had historically had  quality problems in production because designs were incomplete. The B-2  bomber program and the C-17 Airlifter program, for example, encountered  major manufacturing problems because they went forward with unstable  designs and relied on inspections to find defects once in production. Since  1996, GAO has recommended several times that DOD adopt a knowledge- based acquisition approach used by leading commercial companies to  develop its weapon systems. Under this approach, high levels of  knowledge are demonstrated at critical decision points in the product  development process, which results in successful product development  outcomes. Systems engineering is a key practice that companies use to  build quality into new products. Companies translate customers’ broad  requirements into detailed requirements and designs, including identifying  requisite technological, software, engineering, and production capabilities.  Systems engineering also involves performing verification activities,  including testing, to confirm that the design satisfies requirements.  Products borne out of a knowledge-based approach stand a significantly  better chance to be delivered on time, within budget, and with the  promised capabilities. Related GAO products, listed at the back of this  report, provide detailed information about the knowledge-based approach.
section_title: DOD Weapon Systems Experience Quality Problems Due to Prime Contractors’ Inconsistent Practices
paragraphs: Although major defense contractors have adopted commercial quality  standards in recent years, quality and reliability problems persist in DOD  weapon systems. On the 11 weapon systems GAO reviewed, these  problems have resulted in billions of dollars in cost overruns, years of  schedule delays, and reduced weapon system availability. Prime  contractors’ poor systems engineering practices related to requirements  analysis, design, and testing were key contributors to these quality  problems. We also found problems with manufacturing and supplier  quality that contributed to problems with DOD weapon systems. Senior  officials from the prime contractor companies we contacted said that they  agreed with our assessment of the causes of the quality problems of  weapon system programs we reviewed and that disciplined processes help  improve overall quality.
subsections: section_title: Case Studies Illustrate Impact of DOD Weapon System Quality Problems
paragraphs: Quality problems caused significant cost and/or schedule delays in the 11  weapon systems we reviewed. Figure 1 shows the types of problems we  found and the resulting impacts. Appendix II contains detailed information  about each of the programs’ quality problems. Quality problems occurred  despite the fact that each of the prime contractors for these programs is  certified to commercial quality standards and most provided us with  quality plans that address systems engineering activities such as design, as  well as manufacturing, and supplier quality. However, quality problems in  these areas point to a lack of discipline or an inconsistency in how prime  contractors follow through on their quality plans and processes.
section_title: Lack of Systems Engineering Discipline Early in Programs Leads to Significant Quality Problems Later
paragraphs: GAO’s past work has identified systems engineering as a key practice for  ensuring quality and achieving successful acquisition outcomes. Systems  engineering is a sequence of activities that translates customer needs into  specific capabilities and ultimately into a preferred design. These activities  include requirements analysis, design, and testing in order to ensure that  the product’s requirements are achievable and designable given available  resources, such as technologies. In several of the DOD weapon programs  we reviewed, poor systems engineering practices contributed to quality  problems. Examples of systems engineering problems can be found on the  Expeditionary Fighting Vehicle, Advanced Threat Infrared  Countermeasure/Common Missile Warning System, and Joint Air-to- Surface Standoff Missile programs.
Design problems have hampered the development of the Marine Corps’  Expeditionary Fighting Vehicle. The system, built by General Dynamics, is  an amphibious vehicle designed to transport troops from ships offshore to  land at higher speeds and from farther distances than its predecessor.  According to program officials, prime contractor design and engineering  changes were not always passed to suppliers, resulting in supplier parts  not fitting into assemblies because they were produced using earlier  designs. Systems engineering problems have also contributed to poor  vehicle reliability, even though reliability was a key performance  parameter. Consequently, the prime contractor was only able to  demonstrate 7.7 hours between mission failures, which was well short of  the 17 hours it needed to demonstrate in pre-production testing.  Subsequently, the vehicle’s development phase has been extended.  Program officials estimate that this extension, which will primarily focus  on improving reliability, will last an additional 4 years at an estimated cost  of $750 million.
For several other weapon systems, inadequate testing was another  systems engineering problem. The Army’s Advanced Threat Infrared  Countermeasure/Common Missile Warning System program, developed by  BAE Systems, is designed to defend U.S. aircraft from advanced infrared- guided missiles. Reliability problems related to the Advanced Threat  Infrared Countermeasure jam head forced the Army to initiate a major  redesign of the jam head in fiscal year 2006, and fielding of the subsystem  has been delayed until fiscal year 2010.  According to a prime contractor  official, the reliability problems were caused, at least in part, by  inadequate reliability testing. Likewise, the Joint Air-to-Surface Standoff  Missile program, developed by Lockheed Martin, has experienced a  number of flight test failures that have underscored product reliability as a  significant problem. Ground testing, which prime contractor officials said  could have identified most of the failure modes observed in flight testing,  did not occur initially. Prime contractor officials indicated that ground  testing was not considered necessary because the program was a spin-off  of a previous missile program and there was an urgent need for the new  missile. As a result of the test failures, the program has initiated a  reliability improvement effort that includes ground and flight testing.  A  program official reported that the cost of reliability improvements for  fiscal years 2006 and 2007 totaled $39.4 million.
section_title: Manufacturing Problems Are Often Caused by Lack of Process Controls
paragraphs: GAO’s past work addresses the importance of capturing manufacturing  knowledge in a timely manner as a means for ensuring that an  organization can produce a product within quality targets. Prime  contractor activities to capture manufacturing knowledge should include  identifying critical characteristics of the product’s design and then the  critical manufacturing processes to achieve these characteristics.  Once  done, those processes should be proven to be in control prior to  production. This would include making work instructions available,  preventing and removing foreign object debris in the production process,  and establishing criteria for workmanship. However, prime contractors’  lack of controlled manufacturing processes caused quality problems on  several DOD weapon programs, including the F-22A and LPD 17 programs.
The F-22A, a fighter aircraft with air-to-ground attack capability being built  by Lockheed Martin, entered production with less than 50 percent of the  critical manufacturing processes in control. In 2000, citing budgetary  constraints and specific hardware quality problems that demanded  attention, the Air Force abandoned its efforts to get manufacturing  processes in control prior to the start of production. Subsequently, the  contractor experienced a scrap, rework, and repair rate of about 30  percent on early-production aircraft.  The contractor also experienced  major problems with the aircraft canopy. According to program officials,  the aircraft uses a first-of-a-kind canopy, with an external metallic stealth  layer. The contractor did not bring its manufacturing processes in control  and the canopy cracked near the mounting holes. This problem was  discovered in March 2000 and temporarily grounded the flight test aircraft.  In addition, in 2006 a pilot was trapped in an F-22A for 5 hours when a  defective activator prevented him from opening the canopy. According to  the Air Force, when production began in 2001, the prime contractor should  have been able to demonstrate that the F-22A could achieve almost 2 flying  hours between maintenance. However, at that time, the contractor could  demonstrate only about 40 minutes. Six years later, the contractor  increased the flying hours to 97 minutes mean time, short of the Air  Force’s current 3-hour requirement. The program now has budgeted an  additional $400 million to improve the aircraft’s reliability and  maintainability.
Northrop Grumman, the prime contractor for the LPD 17, the first ship of a  new class of amphibious transport dock ships, delivered the ship to the  Navy in 2005 with many quality problems resulting from poor  manufacturing practices. For example, the program experienced problems  with non-skid coating applications because the company did not keep the  boat surface free from dirt and debris when applying the coating, which  caused it to peel. As of late 2007, the problem was not fixed.  In addition,  the ship encountered problems with faulty welds on piping used in some  of the ship’s hydraulic applications. According to the prime contractor,  they could not verify that welds had been done properly. This problem  required increased rework to correct the problems and reinspect all the  welds.  Had the problem not been discovered and weld failure had  occurred, the crew and the ship could have been endangered. These  problems, as well as many others, contributed to a 3-year delay and cost  increase of $846 million in delivering the ship to the Navy.  In June 2007,  the Secretary of the Navy sent a letter to the Chairman of the Board of  Northrop Grumman expressing his concerns about the contractor’s ability  to construct and deliver ships that meet Navy quality standards and to  meet agreed-to cost and schedule commitments.
section_title: Supplier Quality Problems Can Result in Higher Product Cost
paragraphs: Management of supplier quality is another problem area for DOD weapon  systems. Supplier quality is particularly important because more that half  of the cost of a weapon system can be attributed to material received by  the prime contractor from its supplier base.  While DOD prime contractors  told us that they manage and control the quality of parts and material they  receive from their suppliers with the help of performance reviews and  process audits, we found supplier quality problems on seven of the  weapon systems we reviewed.  Two examples are the Wideband Global  SATCOM and Patriot Advanced Capability-3 programs.
Boeing Integrated Defense Systems is the prime contractor for the Air  Force and Army’s Wideband Global SATCOM communications satellite.  Boeing Integrated Defense Systems discovered that one of its suppliers  had installed certain fasteners incorrectly. As a result, 1,500 fasteners on  each of the first three satellites had to be inspected or tested, and 148  fasteners on the first satellite had to be reworked. The DOD program  office reported that the resulting 15-month schedule slip would add  rework and workforce costs to the program and delay initial operating  capability by 18 months. A prime contractor official estimated the cost to  fix the problem was about $10 million.
In 2006, a supplier for the Patriot Advanced Capability-3 program, a long- range system that provides air and missile defense for ground combat  forces, accepted non-conforming hardware for a component for the  missile’s seeker. The seeker contractor had to re-inspect components and  some were returned for rework. As a result of this and other problems  involving poor workmanship and inadequate manufacturing controls, the  supplier facility was shut down for 7 months, delaying delivery of about  100 missiles.
section_title: Prime Contractors’ Observations on Quality
paragraphs: We met with senior quality officials at the prime contractor companies we  included in this review to discuss the problems we found.  For the most  part, they agreed with our assessment, and that the discipline with which a  company implements its processes is a key contributor to quality  outcomes. The officials discussed the importance of quality and how they  are attempting to improve quality across their companies. This includes  the use of Six Sigma, a tool for measuring defects and improving quality,  as well as independent program reviews and improving design processes.
The senior quality officials also identified factors they believe affect the  quality of DOD weapon systems, including insufficient attention to  reliability by DOD during development and the prime contractor’s lack of  understanding of weapon system requirements, including those for testing.
section_title: Leading Commercial Companies Use Disciplined Quality Management Practices
paragraphs: While there are similarities between the quality management practices of  DOD prime contractors and leading commercial companies in our review,  the discipline with which leading companies implement their practices  contributes to the high quality of their products. According to company  officials we contacted, reliability is a paramount concern for them because  their customers demand products that work, and the companies must  develop and produce high-quality products to sustain their competitive  position in the marketplace. Leading commercial companies use  disciplined, well-defined, and institutionalized practices for (1) systems  engineering to ensure that a product’s requirements are achievable with  available resources, such as technologies; (2) manufacturing to ensure that  a product, once designed, can be produced consistently with high quality  and low variability; and (3) supplier quality to ensure that their suppliers  are capable of delivering high-quality parts. These practices, which were  part of the companies’ larger product development processes, and other  tools such as Six Sigma, provided an important foundation for producing  quality products and continually improving performance.
subsections: section_title: Adherence to Systems Engineering Practices Leads to Clear Requirements and Reliable Designs
paragraphs: Several of the companies we met with discussed how they use systems  engineering as a key practice for achieving quality outcomes. As part of  Siemens Medical Solutions’ standard product development process, the  company validates that product requirements are sufficiently clear,  precise, measurable, and comprehensive. They ensure that requirements  address quality, including requirements for reliability and readiness prior  to making a commitment to developing and building a new product.  Officials with Boeing Commercial Airplanes say they have shifted their  view of quality into a more proactive approach, which includes a focus on  “mistake-proofing” designs so that they can be assembled only one way.  To help assess the producibility of critical parts designs, the company has  also developed a tool that rates different attributes of the design, including  clarity of engineering requirements, consequences of defects on  performance or manufacturability, and verification complexity. Company  officials say they use the tool’s ratings to modify designs to ensure that  parts will be less prone to manufacturing and assembly error, and that its  use has resulted in lower costs for scrap, rework, and repair and fewer  quality problems.
Space Systems/Loral also relies on well-defined and disciplined processes  to develop and produce satellites. Because the company’s customers  expect satellites to perform for up to 15 years, product reliability is  paramount and company officials say that using systems engineering to  design reliability into a satellite is essential. As part of its systems  engineering activities, the company performs reliability assessments to  verify that satellite components and subsystems will meet reliability  requirements and to identify potential hardware problems early in the  design cycle. Space Systems/Loral officials also discussed testing and its  importance to developing products. For significant new product  developments, Space Systems/Loral employs highly accelerated life testing  to find weak links in a design and correct them to make the product more  robust before going into production. As a result of the company’s  disciplined quality management practices, new satellite components— such as lithium-ion batteries, stationary plasma thrusters, and a satellite  control system—have over 80 million hours of operation in orbit with only  one component failure, according to company data.
section_title: Effective Manufacturing Process Controls Reduce Variability and Defects
paragraphs: Several company officials discussed the importance of having controlled  manufacturing processes, and described several approaches to reduce  variability and the likelihood of defects. These approaches greatly increase  the likelihood that a product, once designed, can be produced consistently  and with high quality and low variability. In this way, they reduce waste  and increase a product’s reliability in the field.
Early in its product development process, Cummins, a manufacturer of  diesel and natural gas-powered engines, establishes a capability growth  plan for manufacturing processes. This increases the probability that the  manufacturing process will consistently produce parts that meet  specifications. Prior to beginning production, Cummins completes what it  calls “alpha” and “beta” builds, which are prototypes intended to validate  the product’s design and production processes. Cummins officials noted  that these activities allow them to catch problems earlier in development,  when problems are less costly to fix.
Officials from Kenworth, a manufacturer of heavy- and medium-duty  trucks, described several initiatives it uses to improve manufacturing  process controls. For example, the company has a new electronic system  for process documents. Workers on the manufacturing floor used to rely  on paper installation instructions, and sometimes workers used outdated  instructions. Kenworth officials say that converting to an electronic  system ensures that all workers use the most current process  configuration and reduces rework. For a selected number of processes,  Kenworth has also developed documents that include pictures as well as  engineering specifications to ensure that workers follow the correct  processes, and performs audits to assess whether workers are properly  trained and know where to go if they have questions regarding the  process.
section_title: Companies Hold Suppliers Accountable to Deliver High Quality Parts for the Product
paragraphs: At several of the companies we visited, officials reported that supplier  parts accounted for a substantial amount of the overall product value.  Companies we met with systematically manage and oversee their supply  chain through such activities as regular supplier audits and performance  evaluations of quality and delivery, among other things. Several officials  noted that their supplier oversight focuses on first-tier suppliers, with  limited interaction and oversight of lower-tier suppliers. However,  Kenworth officials said they hold their first-tier suppliers accountable for  quality problems attributable to lower-tier suppliers.
Leading commercial companies we met with set high expectations for  supplier quality. Boeing Commercial Airplanes categorizes its suppliers by  rates of defective parts per million. To achieve the highest rating level, a  supplier must exhibit more than 99 percent part conformance, and  company officials said they have been raising their supplier quality  expectations over time. The organization has taken steps to reduce the  number of direct suppliers and retain higher-performing suppliers in the  supply base. Similarly, suppliers of major components for Siemens  Medical Solutions’ ultrasound systems must provide conforming products  98 percent of the time, and the company will levy financial penalties  against suppliers that do not meet this standard. Other companies also  financially penalized suppliers for providing nonconforming parts.
section_title: Disciplined Processes and Continuous Quality Improvement Are a Focus at Several Companies
paragraphs: Several company officials discussed how a focus on improving product  development processes and product quality served as the foundation for  their systems engineering, manufacturing, and supplier quality practices.  Officials with Space Systems/Loral discussed how they adopted a more  disciplined product development process following quality problems in the  1990s with some of its satellites. This included creating companywide  product development processes, adopting a formal program that  institutionalized an iterative development process, and implementing strict  documentation requirements and pass/fail criteria. The company also  established an oversight organization to ensure that processes are  followed. As a result, the first-year failure rate for Space Systems/Loral’s  satellites decreased by approximately 50 percent from 2000 through 2006.
Likewise, Cummins officials told us that quality problems following the  initial release of their ISX engine were a major factor in the  implementation of their current product development process. This  includes review gates to ensure process compliance and management  reviews that use knowledge-based approaches for evaluating projects.
Cummins and Kenworth also use tools such as Six Sigma to define,  measure, analyze, control, and continually improve their processes. For  example, Cummins applies Six Sigma to its technology development,  design, and production activities. The company also expects its critical  suppliers to implement Six Sigma programs to improve quality and  customer satisfaction. As a result of implementing initiatives such as Six  Sigma, Cummins officials reported that the company’s warranty costs have  declined substantially in the last several years. Kenworth also uses Six  Sigma to drive efficiencies into the organization’s work processes,  particularly in the design phase of new product development and in  controlling manufacturing processes. Kenworth requires its first-tier  suppliers to participate in a Six Sigma program. Company officials  estimated that Six Sigma projects saved its Chillicothe, Ohio, facility  several million dollars in 2006.
In addition, each of the commercial companies we met with collected and  used data to measure and evaluate their processes and products. This  helps them gauge the quality of their products and identify areas that need  improvement. For example, Cummins tracks warranty costs as a measure  of product quality, while Siemens Medical Solutions measures  manufacturing process yields for its ultrasound systems.
section_title: Different Environments Create Different Incentives to Improve Quality
paragraphs: The quality problems in our case studies and the practices that relate to  them—whether systems engineering, manufacturing, or supplier quality  practices—are strongly influenced and often the result of larger  environmental factors.  DOD’s acquisition environment is not wholly  conducive to incentivizing prime contractors to efficiently build  high- quality weapon systems—ones that perform as expected, can be depended  on to perform when needed, and are delivered on time and within cost  estimates. During systems development, DOD usually pays for a  contractor’s best efforts, which can include efforts to achieve overly  optimistic requirements. In such an environment, seeking to achieve  overly optimistic requirements along with a lack of oversight over the  development process contributes to quality problems. In contrast,  commercial companies we visited operate in an environment that requires  their own investment of significant funds to develop new products before  they are able to sell them and recoup that investment.  This high-cost  environment creates incentives for reasonable requirements and best  practices, as well as continuous improvement in systems engineering,  manufacturing, and supplier quality.
subsections: section_title: DOD’s Environment
paragraphs: DOD uses cost-reimbursement contracts with prime contractors for the  development of its weapon systems.  In this type of contract arrangement,  DOD accepts most of the financial risks associated with development  because of technical uncertainties. Because DOD often sets overly  optimistic requirements for new weapon systems that require new and  unproven technologies, development cycles can take up to 15 years.  The  financial risk tied to achieving these requirements during development is  not borne by the contractor in this environment, but by the government.   This environment provides little incentive for contractors to utilize the  best systems engineering, manufacturing, and supplier quality practices  discussed earlier in this report to ensure manageable requirements, stable  designs, and controlled manufacturing processes to hold costs down.  Finally, DOD’s quality organizations, which collect information about  prime contractors’ quality systems and problems, provide limited oversight  of prime contractor activities and do not aggregate quality data in a  manner that helps decision makers assess or identify systemic quality  problems.
section_title: Overly Optimistic Requirements Hamper Good Quality Outcomes
paragraphs: DOD’s ability to obtain a high-quality weapon system is adversely  impacted by an environment where it both (1) assumes most of the  financial risks associated with technical or cost uncertainties for the  systems development and (2) sets requirements without adequate systems  engineering knowledge.  Without requirements that have been thoroughly  analyzed for feasibility, development costs are impossible to estimate and  are likely to grow out of control.
DOD typically assumes most of the financial risk associated with a new  weapon system’s development by establishing cost reimbursement  contracts with prime contractors. In essence, this means that prime  contractors are asked to give their best effort to complete the contract and  DOD pays for allowable costs, which often includes fixing quality  problems experienced as part of the effort.  As stated earlier, these  problems can cost millions of dollars to fix.  For example, DOD as the  customer for the Expeditionary Fighting Vehicle signed a cost  reimbursement contract with the prime contractor, General Dynamics, to  develop a new weapon system that would meet performance and  reliability requirements that had not yet been adequately informed by  systems engineering analysis.  Once General Dynamics performed a  detailed requirements analysis, it informed DOD that more resources  would be needed to meet the key reliability requirement established  earlier.  DOD decided not to invest the additional money at that time.   However, when the vehicle was unable to meet its reliability goal prior to  moving into production, DOD eventually decided to invest an additional  $750 million into its development program to meet the reliability goal.
Often DOD enters into contracts with prime contractors before  requirements for the weapon systems have been properly analyzed.  For  example, in March 2007 we reported that only 16 percent of the 62 DOD  weapon system programs we reviewed had mature technologies to meet  requirements at the start of development.  The prime contractors on these  programs ignored best systems engineering practices and relied on  immature technologies that carry significant unknowns about whether  they are ready for integration into a product. The situation is exacerbated  when DOD adds or changes requirements to reflect evolving threats. Prime  contractors must then spend time and resources redesigning the weapon  system, flowing down the design changes to its suppliers, and developing  new manufacturing plans.  In some cases, special manufacturing tools the  prime contractor thought it was going to use might have to be scrapped  and new tooling procured.
Lack of detailed requirements analysis, for example, caused significant  problems for the Advanced Threat Infrared Countermeasure/Common  Missile Warning System program. Prior to 1995, the services managed  portions of the program separately. Then, in 1995, DOD combined the  efforts and quickly put a developer on contract.  This decision resulted in  significant requirements growth and presented major design and  manufacturing difficulties for the prime contractor.  It took over a year to  determine that the tactical fixed-wing aircraft requirements were  incorrect.  The extent of the shortfall, however, did not become evident  until the critical design review and numerous changes were required in the  contract statement of work.  More than 4 years after the system’s critical  design review, the sensor units were built in prototype shops, with  engineers only then trying to identify critical manufacturing processes.   Further, sensor manufacturing was slowed by significant rework, and at  one point was halted while the contractor addressed configuration control  problems.  The Navy and Air Force, which required the system for fixed- wing aircraft, dropped out of the program in 2000 and 2001, respectively.
Ultimately, quality is defined in large part by reliability.  But, in DOD’s  environment, reliability is not usually emphasized when a program begins,  which forces the department to fund more costly redesign or retrofit  activities when reliability problems surface later in development or after a  system is fielded. The F-22A program illustrates this point.  Because DOD  as the customer assumed most of the financial risk on the program, it  made the decision that system development resources primarily should be  focused on requirements other than reliability, leading to costly quality  problems. After 7 years in production, the Air Force had to budget an  additional unplanned $400 million for the F-22A to address numerous  quality problems and help the system achieve its baseline reliability  requirements.
section_title: Oversight of Development Programs Could Be Strengthened
paragraphs: DOD oversight of prime contractor activities varies and has decreased as  its quality assurance workforce has decreased.  Weapon system progress  reviews at key decision points are a primary means for DOD to oversee  prime contractor performance in building high-quality systems, but they  are not used consistently across programs.  The purpose of the reviews is  to determine if the program has demonstrated sufficient progress to  advance to the next stage of product development or to enter production.   The department has developed decision criteria for moving through each  phase of development and production; and DOD’s acquisition executive  has the authority to prevent programs from progressing to later stages of  development if requisite knowledge has not been attained.  Unfortunately  most programs are allowed to advance without demonstrating sufficient  knowledge. For example, in our recent review of 62 DOD weapon systems,  we found that only 27 percent of the programs demonstrated that they had  attained a stable design at the completion of the design phase.
In addition, as a result of downsizing efforts over the past 15 years, DOD’s  oversight of prime contractor and major supplier manufacturing processes  varies from system to system.  DOD quality officials stated that they have  had to scale back on the amount of oversight they can provide, focusing  only on the specific areas that the weapon system program managers ask  them to review.  It is unclear what impact the reduction in quality  assurance specialists and the reduction of oversight has had on the  department’s ability to influence quality outcomes.  However, in the case  of the Advanced SEAL Delivery System, a lapse in effective management  oversight exercised by both the government and contractor contributed to  very late discovery of costly quality problems.  DOD quality organizations  such as the Defense Contract Management Agency do capture a significant  amount of information electronically about the quality of DOD weapon  systems through audits and corrective action reports. They collect quality  data on a program by program basis and share information about certain  types of deficiencies and nonconforming parts they found.  While the  organizations are looking for additional opportunities to share  information, they do not currently aggregate and consolidate the  information in a manner that would allow the department to determine the  overall quality of products it receives from prime contractors or to identify  quality related systemic problems or trends with its prime contractors.
section_title: Commercial Environment
paragraphs: Commercial companies must develop and deliver high-quality, highly  capable products to markets on-time or suffer financial loss. The  companies face competition and, therefore, their customers can choose  someone else’s products when they are not satisfied.  It is this  environment that incentivizes manufacturers to implement and use best  practices to improve quality and reduce cost while delivering on-time.  Commercial customers must set achievable product requirements for their  manufacturers that they know will result in a reliable, high-quality, and  desirable product that can be delivered on-time. Manufacturers then get  their key manufacturing processes in control to reduce inconsistencies in  the product.  Commercial customers understand the need to monitor and  track manufacturer and supplier quality performance over time to  determine which companies they want to do business with in the future or  to identify problem areas that need to be corrected.
Commercial customers we visited—American Airlines and Intelsat— expect to operate their products for 30 and 15 years, respectively.  The  companies focus a great deal of attention on setting performance and  reliability goals that manufacturers like Boeing Commercial Airplanes and  Space Systems/Loral must meet in order for them to purchase their  products.  This provides a strong, direct incentive for manufacturers and  their customers to ensure that requirements are clear and achievable with  available resources, including mature technologies, before the  manufacturer will invest in a product’s development.  For example,  Intelsat expects its satellites to be available at least 99.995 percent of the  time.  To meet this goal, Intelsat expects its manufacturers to use mature  technologies and parts where the reliability is already known. There are  several reasons that drive this approach. The most obvious one is that  there is no way to fix mechanical problems once a satellite has been  launched.  Another reason is that the company must credit television  networks, telephone companies, or cable companies for any loss of  service.  The company also insures their satellite for launch plus the first  year of in-orbit service. Having a proven record of in-orbit performance  and using reliable and flight proven technology are two important factors  that help the company get favorable terms from the insurance  underwriters.  And, the company does not want to spend a large sum of  money for a replacement satellite prior to its design life since it will  negatively impact the company’s financial performance.
In the commercial environment, manufacturers are motivated to develop  and provide high-quality products because their profit is tied to customer  expectations and satisfaction. For example, American Airlines makes an  initial payment to Boeing Commercial Airplanes when it places an order  for new aircraft, but will not make final payment until it is satisfied that  their requirements have been met.
In an another example, Cummins officials discussed how they were  motivated to adopt more disciplined product development processes  following the development effort for one of its highest selling engines, in  the late 1990s. According to company officials, the design requirements  were unstable from the start of development.  They were changed and  added to as development progressed, often without the benefit of timely  and disciplined requirements analysis to ensure they could be met for the  estimated investment cost.  There were conflicting requirements (weight,  size, performance, and fuel economy) that made development difficult. In  addition, Cummins did not pay enough attention to reliability, focusing  instead on weight and power considerations.  As a result, development  costs were higher than expected and, once the engine was sold, customers  experienced less than expected. A Cummins official reported that the  company found itself in an “intolerable” position with customers who  were becoming increasingly dissatisfied.
This significant event, in which Cummins lost customer confidence,  caused the company to examine its product development processes. The  result of this examination was an improved product development process  that requires a more cross functional and data-based approach to new  development programs. The improvements resulted in better analysis and  understanding of customer requirements leading to resource allocations  before beginning new programs. Cummins invested in both customer  satisfaction and the development and support of its products.  This  investment provided the motivation to adopt a more disciplined product  development approach for the production of high-quality products for its  customers.
Intelsat officials told us it makes progress payments to its manufacturers  throughout development and production.  However, the company holds  about 10 to 20 percent of the contract value to award to the manufacturer  after a satellite is successfully launched.  According to company officials,  the 10 to 20 percent is paid to the manufacturer over the expected life of  the satellite, which is typically 15 years, when the satellite performs as  expected.
The commercial companies also all capture information about their  manufacturing processes and key suppliers’ quality.  However, unlike  DOD, they use the information when making purchasing decisions and  determining how best to structure contracts to incentivize good quality  outcomes.  For example, in some cases Intelsat does not allow  manufacturers to use certain suppliers whose parts do not meet specified  reliability goals.  In addition, Intelsat may include clauses in its contracts  that require a manufacturer to conduct periodic inspections of particular  suppliers.
section_title: DOD Efforts to Improve Acquisition Outcomes
paragraphs: DOD has long recognized its acquisition problems and has initiated  numerous improvement efforts over the years to address them. A recent  set of initiatives are highlighted by the Under Secretary of Defense for  Acquisition, Technology and Logistics in DOD’s Defense Acquisition  Transformation and Program Manager Empowerment and Accountability  reports to Congress. Our analysis indicates that while none of the  initiatives is aimed solely at improving the quality of DOD weapon systems  or improving prime contractor quality practices, they could address some  of the problems identified in this report, particularly the ones that improve  the DOD requirements-setting process and limit requirements growth  during development. A brief description of the initiatives is included  below.
Concept Decision Reviews: DOD is pilot-testing a concept decision  reviews program to provide a better framework for strategic  investment decisions. A Concept Decision Committee composed of  senior DOD officials is applying the reviews to four pilot programs— the Joint Lightweight Tactical Mobility program, the Integrated Air and  Missile Defense program, the Global Strike Raid Scenario, and the  Joint Rapid Scenario Generation program. A key aspect of the pilot  programs is the early involvement and participation of systems  engineering prior to concept decision. DOD expects this to provide  decision makers better insight for setting firm requirements early,  assessing technology options, considering alternative acquisition  strategies, ensuring that new technology will mature in time to meet  development and delivery schedules, and delivering systems with  predictable performance to the warfighter.
Time-Defined Acquisition: Under the time-defined acquisition  initiative, DOD plans to use such criteria as technology maturity, time  to delivery, and requirement certainty to select the appropriate  acquisition approach to provide a needed capability. The department  envisions using a different acquisition approach, depending on whether  a capability can be fielded in 2 years or less, more than 2 years to less  than 4 years, or more than 4 years. In September 2006, the Under  Secretary of Defense for Acquisition, Technology and Logistics stated  that he anticipated the time-defined acquisition approach would  facilitate better overall cost control and more effective use of total  available resources.
Configuration Steering Boards: In July 2007, the Under Secretary of  Defense for Acquisition, Technology and Logistics directed the  establishment of Configuration Steering Boards for every current and  future acquisition category I program in development.  The boards,  chaired by the service acquisition executive within each of the military  services, are expected to review all requirements changes and  significant technical configuration changes that have the potential to  adversely affect program cost and schedule. Requirement changes are  not to be approved unless funds are identified and schedule impacts  are mitigated. However, the Under Secretary stated in his  announcement of this initiative that such requirements changes would  usually be rejected.
Key Performance Parameters/Key System Attributes: DOD has  added new guidelines and procedures for establishing weapon system  requirements in its Joint Capabilities Integration and Development  System manual. The manual now requires that materiel availability be  included as a key performance parameter for new weapon system  development and that materiel reliability and ownership costs be  included as key system attributes.  Together, these requirements are  aimed at ensuring weapon system sustainment considerations are fully  assessed and addressed as part of the systems engineering process.
Award and Incentive Fees:  DOD recently issued policy  memorandums that reflect a change in policy related to the proper use  of award and incentive fees. The memorandums emphasize the need to  structure award fee contracts in ways that focus DOD and contractor  efforts on meeting or exceeding cost, schedule, and performance  requirements.  The policy memorandums state that award fees should  be linked to desired outcomes and payments should be commensurate  to contractor performance. It also provides guidelines for how much  contractors will be paid for excellent performance, satisfactory  performance, and less than satisfactory performance.
While these initiatives are not directly linked together, they have the  potential to help DOD implement some of the leading commercial  practices we have highlighted in the past. In particular, they could help the  Under Secretary of Defense for Acquisition, Technology and Logistics  ensure that DOD has a better match between warfighter needs and funding  at the start of weapon system development and that technology,  engineering, and production knowledge is properly considered at that  time. They can also help control requirements changes and requirements  growth, which can adversely affect system quality during development.  The initiatives are still new and, in the case of concept decision reviews,  small in scope; therefore, their effectiveness may not be known for some  time.
section_title: Conclusions
paragraphs: DOD has developed policies that address the need for setting achievable  requirements, adopting commercial quality standards, using good systems  engineering practices, and overseeing supplier quality. However, DOD still  has difficulty acquiring high-quality weapon systems in a cost-efficient and  timely manner.  While many problems are caused by poor prime  contractor practices related to systems engineering, manufacturing, and  supplier quality, an underlying cause lies in the environment. DOD  typically assumes most of the financial risk associated with development  of complex systems. However, risks associated with this situation are  exacerbated because DOD generally enters into development contracts  without demonstrated knowledge or firm assurance that requirements are  achievable, which too often result in inefficient programs and quality  problems.
DOD can learn from leading commercial companies in the way they deal  with risk and ensure quality in their products.  Because commercial  companies invest their own money in product development and recoup  that investment when their customers buy the finished good, they put a  new product’s requirements to the test with disciplined systems  engineering practices before they commit to a large investment to develop  it. If a highly valued requirement cannot be demonstrated as achievable  through systems engineering, it is deferred to a subsequent product  variation or to another program. Moreover, and very importantly,  companies do not shortcut essential quality practices that ensure process  controls and high supplier quality, including collecting and analyzing  quality data. Like commercial companies, DOD must demand appropriate  knowledge about requirements and make hard decisions about program  risk before it initiates costly investments.
Improvements in the way DOD uses existing tools to analyze requirements  during development, along with potential results of some of the initiatives  it has underway, can help reduce quality risks, and address some of the  long-standing acquisition problems it faces. Although the initiatives are  new and in the case of the concept decision reviews, small in scope, they  are a good first step toward the department setting more realistic  requirements and time frames for weapon system development. Additional  oversight could help ensure that prime contractors can meet requirements  with given resources, such as funding and technologies, prior to DOD  entering into a development contract. In addition, continued leadership  from the Under Secretary of Defense for Acquisition, Technology and  Logistics and a combination of actions from both DOD and prime  contractors are needed to make these improvements and get the most  from its planned $1.5 trillion investment in new weapons programs.
section_title: Recommendations for Executive Action
paragraphs: To ensure that the department is taking steps to improve the quality of  weapon systems, we recommend that the Secretary of Defense take the  following actions related to recent initiatives highlighted in DOD’s Defense  Acquisition Transformation and Program Manager Empowerment and  Accountability reports to Congress to improve its focus on setting  achievable requirements and oversight:
As a part of the concept decision review initiative, have   contractors perform more detailed systems engineering analysis to  develop sound requirements before DOD selects a prime  contractor for the systems development contract, which would  help ensure that weapon system requirements, including those for  reliability, are achievable with given resources.
Establish measures to gauge the success of the concept decision  reviews, time-defined acquisition, and configuration steering board  initiatives and properly support and expand these initiatives where  appropriate.
To better assess the quality of weapon system programs and prime  contractor performance, DOD needs to obtain and analyze more  comprehensive data regarding prime contractors and their key suppliers.  Therefore, we also recommend that the Secretary of Defense direct the  Defense Contract Management Agency and the military services to:  Identify and collect data that provides metrics about the  effectiveness of prime contractors’ quality management system and  processes by weapon system and business area over time and
Develop evaluation criteria that would allow DOD to score the  performance of prime contractors’ quality management systems  based on actual past performance, which could be used to improve  quality and better inform DOD acquisition decision makers.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report.  DOD  partially concurred with each of the recommendations.  DOD’s comments  appear in appendix III.
In its comments, DOD partially concurred with the draft recommendation  that, as part of its concept decision review initiative, prime contractors  should complete systems engineering analysis prior to entering a  development contract. The department stated that the recommendation  was vague.  DOD noted that it conducts systems engineering planning  prior to entering into a development contract and that prime contractors  conduct more detailed systems engineering analysis afterwards.   Moreover, DOD noted that systems engineering is a continuous  government-performed activity at the heart of any structured development  process that proceeds from concept to production.  The concept decision  review initiative, in particular, considers fundamental systems engineering  issues such as technology and integration and manufacturing risk before  the concept decision review.
To address DOD’s concern that our recommendation was too vague, we  modified it to add more detail. Specifically, as part of the concept decision  review initiative, we recommend that contractors that are competing for  the systems development contract provide DOD more detailed systems  engineering requirements analysis to be completed before a systems  development contract is awarded. This would help ensure that  requirements are clear and reasonable before DOD enters into a  development contract. We understand that currently DOD conducts  systems engineering planning prior to entering a development contract  with prime contractors and that prime contractors conduct a more  thorough systems engineering analysis afterwards.  However, because our  work has found that many DOD systems development efforts have been  hampered by poorly defined or poorly understood requirements, we  believe that DOD should test, through the concept decision initiative,  paying contractors to complete a more thorough systems engineering  analysis prior to entering into a development contract.  This would give  the department the benefit of more knowledge when finalizing  requirements and provide an opportunity for DOD to set requirements that  can be met in a well-defined time frame, which could reduce the  department’s risk exposure in cost reimbursement contracts used for  development.  In addition, it would better position DOD to place more  accountability on the winning contractor to meet the desired requirements  within cost and schedule estimates.
DOD also partially concurred with the recommendation to establish  measures to gauge the success of the concept decision reviews, time- defined acquisition, and configuration steering board initiatives and  properly support and expand these initiatives where appropriate.  In its  response, DOD stated that changes to the concept decision review and  time-defined acquisition initiatives are being considered and any changes  would be reflected in an update to DOD Instruction 5000.2.  DOD also  stated that the configuration steering board initiative is being implemented  consistent with its policy.
We are encouraged by the potential changes that could result from  successful implementation of the concept decision reviews, time-defined  acquisition, and configuration steering board initiatives.  We believe that  these three initiatives are aimed at addressing several of DOD’s systemic  problems that impact weapon system quality and that the department  should not lose sight of these initiatives. While the initiatives are new and  untested in practice, acquisition history tells us that these policy changes  alone will not be sufficient to change outcomes.  We have found that  measures to gauge success can help facilitate senior-level oversight that is  needed to bring about significant change within an organization. We,  therefore, believe this recommendation remains valid.
DOD partially concurred with the recommendation for the Defense  Contract Management Agency and military services to identify and collect  data that provides metrics about the effectiveness of prime contractors’  quality management systems and processes by weapon system and  business area over time.  In its response, DOD stated that the Defense  Contract Management Agency is in the process of identifying and will  eventually collect data that could be used to determine the effectiveness of  prime contractors’ quality management systems.  However, DOD stated  that the added expense of capturing data by weapon system and business  area does not seem warranted at this time.  Further it commented that  there is no need for the military services to engage in a similar effort to the  Defense Contract Management Agency, since the agency is working in  cooperation with the military services.
We are encouraged by the Defense Contract Management Agency’s efforts  to identify and collect data on prime contractor quality management  activities on a broad scale.  As we noted in the report, this is a practice  used by leading commercial companies we visited.  During our review, the  agency could only provide data on a weapon system by weapon system  basis.  We believe that data should be captured on both a weapon system  and prime contractor basis and that the added expense of including data  by weapon system is likely minimal, given that it is already being collected  that way. Considering that DOD plans to invest about $1.5 trillion (in 2007  dollars) in its current portfolio of major weapon systems, we believe it  would be valuable for DOD to know how the companies and business  units responsible for delivering its high-quality weapon systems are  performing as well as the quality associated with individual weapon  systems.  In addition, we believe the military services, particularly the  Navy’s Supervisor of Shipbuilding organization, which is responsible for  overseeing contractor activities for shipbuilding, should identify and  collect similar data so that information collected is consistent and can be  used for comparison purposes. We, therefore, believe this  recommendation remains valid.
Finally, DOD partially concurred with the recommendation for the  Defense Contract Management Agency and military services to develop  evaluation criteria that would allow DOD to score the performance of  prime contractors’ quality management systems based on actual past  performance. DOD stated that it plans to develop evaluation criteria based  on data the Defense Contract Management Agency plans to collect in the  future.   DOD does not think the military services need to develop a  parallel effort because Defense Contract Management Agency data will be  shared with the military services.
It was not our intent for the military services, the Defense Contract  Management Agency, and the Navy’s Supervisor of Shipbuilding to have  parallel efforts. Rather, we expected that they would work collaboratively  on this effort. Moreover, not only do we believe DOD should know how  well the prime contractors and their respective programs are performing  as noted above, we also believe that DOD should know how well the prime  contractors’ quality management systems are working.  Again, this is a  practice used by leading commercial companies we visited.  We are  encouraged that the Defense Contract Management Agency plans to  develop evaluation criteria that would be used to score prime contractor  quality management systems but believe the department should have a  consistent methodology to be used across DOD. We, therefore, believe this  recommendation remains valid.
We are sending copies of this report to the Secretary of Defense and  interested congressional committees.  We will also make copies available  at no charge on the GAO Web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please contact me at (202) 512-4841 or sullivanm@gao.gov.  Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report compares Department of Defense (DOD) and its large prime  contractors’ quality management policies and practices with those of leading  commercial companies—with a focus on improving the quality of DOD weapon  systems. Specifically, we (1) determined the impact of quality problems on  selected DOD weapon systems and defense contractors’ practices that  contributed to the problems, (2) identified practices used by leading  commercial companies that can be used to improve the quality of DOD weapon  systems, (3) identified problems DOD faces in terms of improving quality, and  (4) identified recent DOD initiatives that could improve quality.
To determine the impact of quality problems on selected DOD weapon  systems and defense contractors’ practices that contribute to the problems,  we selected and reviewed 11 DOD weapon systems with known deficiencies  from each of the military services and identified the quality problems  associated with each deficiency. The 11 were chosen to demonstrate the  types of problems DOD weapon systems experience and to help focus our  discussions with leading commercial companies on aspects of  development that caused DOD major quality problems.  The prime  contractors in charge of developing these systems include six of DOD’s  largest contractors; together, they are involved with a little over $1 trillion, or  about 76 percent, of the $1.5 trillion (in 2006 dollars) DOD plans to spend on  weapon systems in its current portfolio. Systems we reviewed, along with the  prime contractors responsible for developing the systems, are:    Advanced SEAL Delivery System, a battery-powered submarine  funded by the Special Operations Command and developed by  Northrop Grumman;    Advanced Threat Infrared Countermeasure/Common Missile  Warning System, a defense countermeasure system for protection  against infrared guided missiles in flight funded primarily by the Army  and developed by BAE Systems;    Expeditionary Fighting Vehicle, an amphibious and armored  tracked vehicle funded by the Navy for the Marine Corps and  developed by General Dynamics;    F-22A, an air superiority fighter with an air-to-ground attack capability  funded by the Air Force and developed by Lockheed Martin;    Global Hawk, a high-altitude, long endurance unmanned aircraft  funded by the Air Force and developed by Northrop Grumman;    Joint Air-to-Surface Standoff Missile, an air-to-surface missile  funded by the Air Force and developed by Lockheed Martin;    LPD 17, an amphibious transport ship funded by the Navy and  developed by Northrop Grumman;   MH-60S, a fleet combat support helicopter funded by the Navy and  developed by Sikorsky Aircraft;    Patriot Advanced Capability-3, a long-range high-to-medium altitude  missile system funded by the Army and developed by Lockheed Martin;    V-22, a tilt rotor, vertical/short take-off and landing aircraft funded  primarily by the Navy for the Marine Corps and developed jointly by  Bell Helicopter Textron and Boeing Integrated Defense Systems; and    Wideband Global SATCOM, a communications satellite funded by  the Air Force and developed by Boeing Integrated Defense Systems.
To evaluate each of the 11 DOD weapon systems, we examined program  documentation, such as deficiency reports and corrective action reports,  and held discussions with quality officials from DOD program offices, the  prime contractor program office, and either the Defense Contract  Management Agency or the Supervisor of Shipbuilding office where  appropriate. Based on information gathered through documentation and  discussions, we grouped the problems into three general categories:  systems engineering, manufacturing, and supplier quality. When possible,  we identified the impact that quality problems had on system cost,  schedule, performance, reliability, availability, or safety. After completing  our weapon systems reviews, we held meetings with senior quality leaders  at selected prime contractors included in our review to discuss the quality  problems we found and to obtain their views on why the problems  occurred.
To identify practices used by leading commercial companies that can be  used to improve the quality of DOD weapon systems, we selected and  visited five companies based on several criteria: companies that make  products similar to DOD weapon systems in terms of complexity;  companies that have been recognized in quality management literature or  by quality-related associations/research centers for their high-quality  products; companies that have won quality-related awards; and/or  companies that have close relationships with customers when developing  and producing products. We met with these companies to discuss their  product development and manufacturing processes, supplier quality  activities, and the quality of selected products made by these companies.  Much of the information we obtained from these companies is anecdotal,  due to the proprietary nature of the data that could affect their competitive  standing. Several of the companies provided data on specific products,  which they agreed to let us include in this report. The companies we  visited and the products we discussed include:    Boeing Commercial Airplanes, a leading aerospace company and a  manufacturer of commercial jetliners. We met with quality officials in  Seattle, Washington, and discussed the quality practices associated  with the company’s short-to-medium range 737 and extended range 777  aircraft, as well as its new 787 aircraft.
Cummins Inc., a manufacturer of diesel and natural gas-powered  engines for on-highway and off-highway use. We met with quality  officials at their company’s headquarters location in Columbus,  Indiana, and discussed the development and quality of the ISX, a  heavy-duty engine.
Kenworth Truck Company, a division of PACCAR Inc. and a leading  manufacturer of heavy- and medium-duty trucks. We met with quality  officials at its manufacturing plant in Chillicothe, Ohio, which was  named Quality Magazine’s 2006 Large Plant of the Year, to discuss the  development and quality of various large trucks.
Siemens Medical Solutions, a business area within Siemens AG,  which is a global producer of numerous products, including  electronics, electrical equipment, and medical devices. We met with  quality officials at a company facility located in Mountain View,  California, and discussed the division’s quality practices for developing  and manufacturing ultrasound systems such as the Sequoia ultrasound  system.
Space Systems/Loral, one of the world’s premier designers,  manufacturers, and integrators of geostationary satellites and satellite  systems. We met with quality officials at the company’s headquarters in  Palo Alto, California, and discussed the company’s quality practices for  developing satellites such as the Intelsat IX series and iPSTAR  satellites.
To identify problems that DOD must overcome to improve the quality of  weapon systems, we reviewed processes and tools DOD can use to influence  weapon system quality. These include setting requirements, participating in  key decisions during weapon system development and production, using  contracts to incentivize good quality, and overseeing weapon system quality  and prime contractor performance. We examined these processes and tools  for the 11 weapons programs we reviewed and discussed the use of these  processes and tools with acquisition and quality officials from the Office of  the Secretary of Defense, military services, prime contractors, Defense  Contract Management Agency, and Supervisor of Shipbuilding. We also relied  on previous GAO best practices and weapon system reports to identify DOD  actions that contributed to poor quality outcomes. A comprehensive list of  reports we considered throughout our review can be found in the related  products section at the end of this report.
We met with officials at two commercial companies that purchase  products manufactured by two of the leading commercial manufacturers  we included in this review. These companies included:    American Airlines, the largest scheduled passenger airline in the  world, which has purchased aircraft from Boeing Commercial  Airplanes. We met with quality officials at a major maintenance facility  located in Tulsa, Oklahoma.
Intelsat, a leading provider of fixed satellite services for  telecommunications, Internet, and media broadcast companies, which  purchases satellites from all major satellite manufacturers in the United  States and Europe. We met with officials in space systems acquisition and  planning at the company’s headquarters located in Washington, D.C.
Our discussions focused on (1) the companies’ roles in establishing  requirements; (2) the types of contracts they award to manufacturers and  the specificity included in the contracts in terms of quality, reliability, and  penalties; and (3) the amount of oversight they exercise over their  suppliers’ development and manufacturing activities.
To identify recent DOD initiatives that could improve weapon system  quality, we reviewed DOD’s formal response to Sections 804 and 853 of the  John Warner National Defense Authorization Act for Fiscal Year 2007. This  act requires DOD to report to the congressional defense committees on  acquisition reform and program management initiatives. We also met with  senior defense leaders to discuss the implementation status of the  acquisition reform initiatives identified in DOD’s February 2007 and  September 2007 reports to the committees and relied on a previous GAO  report for the implementation status of planned program management  improvements.
We conducted this performance audit from September 2006 to December  2007 in accordance with generally accepted government auditing  standards.  Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives.  We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Quality Problems for 11 DOD Weapon Systems
paragraphs: This appendix summarizes the quality problems experienced by the 11 DOD  weapon systems we reviewed. The problems are categorized as systems  engineering, manufacturing, and/or supplier quality problems. Most of the  programs had problems in more than one of these categories. These  summaries do not address all quality problems experienced on the programs;  rather they emphasize major problems we discussed with officials from the  military services, prime contractors, and the Defense Contract Management  Agency. When possible, we include the direct impact the quality problems had  on the program, the corrective actions the prime contractor or DOD took to  address the problems, and the change in cost estimates and quantities from  the start of program development to the present.
The cost estimates were taken from DOD Selected Acquisition Reports or  were program office estimates and include DOD’s research, development,  test and evaluation (RDT&E) and procurement expenditures on a  particular program. We did not break out the portion of these funds that  were paid to prime contractors versus the amount paid to suppliers. In  addition, the change in cost estimates can be the result of a number of  factors, including the amount paid to fix quality problems, a decision to  procure more weapons, and increased labor rates or material prices.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
paragraphs: Michael Sullivan (202) 512-4841 or sullivanm@gao.gov.
subsections: section_title: Acknowledgments
paragraphs: Key contributors to this report were Jim Fuquay, Assistant Director;  Cheryl Andrew; Lily Chin; Julie Hadley; Lauren Heft; Laura Jezewski;  Andrew Redd; Charlie Shivers, and Alyssa Weir.
section_title: Related GAO Products
paragraphs: Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-07-406SP. Washington, D.C.: March 30, 2007.
Best Practices: Stronger Practices Needed to Improve DOD Technology  Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006.
Best Practices: Better Support of Weapon System Program Managers  Needed to Improve Outcomes. GAO-06-110. Washington, D.C.:   November 1, 2005.
Defense Acquisitions: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.  Washington, D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.:   February 11, 2003.
Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept  Technology Demonstration. GAO-03-52. Washington, D.C.: December 2,  2002.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:   July 15, 2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:   March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:  July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.:   April 26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Programs Implement Best Practices. GAO/NSIAD-99-206.   Washington, D.C.: August 16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162.   Washington, D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisitions: Improved Program Outcomes Are Possible.   GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:   August 26, 1996.