Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality
Released: 2008-02-01T00:00:00

Letter

Background

In general, a quality product is one that is delivered on time, performs as expected, and can be depended on to perform when needed, at an affordable cost. This applies whether the customer is an individual purchasing a simple consumer good, such as a television, a hospital purchasing medical imaging equipment to help doctors treat cancer patients, or DOD purchasing sophisticated weapons for its warfighters to use on the battlefield.

For about 3 decades, DOD based its quality requirements on a military standard known as MIL-Q-9858A, and its quality assurance practices were oriented toward discovering defects through inspections. In 1994, the Secretary of Defense announced that commercial quality standards should replace MIL-Q-9858A. The intent was to remove military-unique requirements that could present barriers to DOD in accessing the commercial supplier base. Currently, responsibilities for quality policy and oversight fall under the Systems and Software Engineering organization, within the Office of the Secretary of Defense.

Over the past 20 years, commercial companies have had to dramatically improve quality in response to increased competition. Many companies moved from inspection-oriented quality management practices—where problems are identified and corrected after a product is produced—to a process in which quality is designed into a product and manufacturing processes are brought in statistical control to reduce defects. Many companies have also adopted commercial quality standards, such as ISO 9001. This standard was developed by the International Organization for Standardization, a non-governmental organization established in 1947 to facilitate the international coordination and unification of industrial standards. Similar to DOD’s MIL-Q-9858A, ISO 9001 includes requirements for controlling a product’s design and development, and production, as well as processes for oversight and improvement. Some industries, such as the automotive and aerospace industries, also have standards specific to their sector based on the ISO 9001. Because supplier parts account for a substantial amount of the material value of many companies’ products, companies may require their suppliers to adopt the same standards.

In practice, DOD and its prime contractors both participate in activities that contribute to weapon system quality. DOD plays a large role in quality when it sets key performance parameters, which are the most important requirements DOD wants prime contractors to focus on during development. For example, if reliability is one of those key performance parameters, then prime contractors are expected to focus on it during weapon system design. Prime contractors employ quality assurance specialists and engineers to assess the quality and reliability of parts they receive from suppliers, as well as the overall weapon system. DOD has its own quality specialists within the Defense Contract Management Agency and the military services, such as the Navy’s Supervisor of Shipbuilding organization. DOD’s quality specialists oversee prime contractors’ design, manufacturing, and supplier management activities; oversee selected supplier manufacturing activities; and conduct final product inspections prior to acceptance.

GAO previously reported on DOD quality practices in 1996. At that time, we reported that numerous weapon system programs had historically had quality problems in production because designs were incomplete. The B-2 bomber program and the C-17 Airlifter program, for example, encountered major manufacturing problems because they went forward with unstable designs and relied on inspections to find defects once in production. Since 1996, GAO has recommended several times that DOD adopt a knowledge- based acquisition approach used by leading commercial companies to develop its weapon systems. Under this approach, high levels of knowledge are demonstrated at critical decision points in the product development process, which results in successful product development outcomes. Systems engineering is a key practice that companies use to build quality into new products. Companies translate customers’ broad requirements into detailed requirements and designs, including identifying requisite technological, software, engineering, and production capabilities. Systems engineering also involves performing verification activities, including testing, to confirm that the design satisfies requirements. Products borne out of a knowledge-based approach stand a significantly better chance to be delivered on time, within budget, and with the promised capabilities. Related GAO products, listed at the back of this report, provide detailed information about the knowledge-based approach.

DOD Weapon Systems Experience Quality Problems Due to Prime Contractors’ Inconsistent Practices

Although major defense contractors have adopted commercial quality standards in recent years, quality and reliability problems persist in DOD weapon systems. On the 11 weapon systems GAO reviewed, these problems have resulted in billions of dollars in cost overruns, years of schedule delays, and reduced weapon system availability. Prime contractors’ poor systems engineering practices related to requirements analysis, design, and testing were key contributors to these quality problems. We also found problems with manufacturing and supplier quality that contributed to problems with DOD weapon systems. Senior officials from the prime contractor companies we contacted said that they agreed with our assessment of the causes of the quality problems of weapon system programs we reviewed and that disciplined processes help improve overall quality.

Case Studies Illustrate Impact of DOD Weapon System Quality Problems

Quality problems caused significant cost and/or schedule delays in the 11 weapon systems we reviewed. Figure 1 shows the types of problems we found and the resulting impacts. Appendix II contains detailed information about each of the programs’ quality problems. Quality problems occurred despite the fact that each of the prime contractors for these programs is certified to commercial quality standards and most provided us with quality plans that address systems engineering activities such as design, as well as manufacturing, and supplier quality. However, quality problems in these areas point to a lack of discipline or an inconsistency in how prime contractors follow through on their quality plans and processes.

Lack of Systems Engineering Discipline Early in Programs Leads to Significant Quality Problems Later

GAO’s past work has identified systems engineering as a key practice for ensuring quality and achieving successful acquisition outcomes. Systems engineering is a sequence of activities that translates customer needs into specific capabilities and ultimately into a preferred design. These activities include requirements analysis, design, and testing in order to ensure that the product’s requirements are achievable and designable given available resources, such as technologies. In several of the DOD weapon programs we reviewed, poor systems engineering practices contributed to quality problems. Examples of systems engineering problems can be found on the Expeditionary Fighting Vehicle, Advanced Threat Infrared Countermeasure/Common Missile Warning System, and Joint Air-to- Surface Standoff Missile programs.

Design problems have hampered the development of the Marine Corps’ Expeditionary Fighting Vehicle. The system, built by General Dynamics, is an amphibious vehicle designed to transport troops from ships offshore to land at higher speeds and from farther distances than its predecessor. According to program officials, prime contractor design and engineering changes were not always passed to suppliers, resulting in supplier parts not fitting into assemblies because they were produced using earlier designs. Systems engineering problems have also contributed to poor vehicle reliability, even though reliability was a key performance parameter. Consequently, the prime contractor was only able to demonstrate 7.7 hours between mission failures, which was well short of the 17 hours it needed to demonstrate in pre-production testing. Subsequently, the vehicle’s development phase has been extended. Program officials estimate that this extension, which will primarily focus on improving reliability, will last an additional 4 years at an estimated cost of $750 million.

For several other weapon systems, inadequate testing was another systems engineering problem. The Army’s Advanced Threat Infrared Countermeasure/Common Missile Warning System program, developed by BAE Systems, is designed to defend U.S. aircraft from advanced infrared- guided missiles. Reliability problems related to the Advanced Threat Infrared Countermeasure jam head forced the Army to initiate a major redesign of the jam head in fiscal year 2006, and fielding of the subsystem has been delayed until fiscal year 2010. According to a prime contractor official, the reliability problems were caused, at least in part, by inadequate reliability testing. Likewise, the Joint Air-to-Surface Standoff Missile program, developed by Lockheed Martin, has experienced a number of flight test failures that have underscored product reliability as a significant problem. Ground testing, which prime contractor officials said could have identified most of the failure modes observed in flight testing, did not occur initially. Prime contractor officials indicated that ground testing was not considered necessary because the program was a spin-off of a previous missile program and there was an urgent need for the new missile. As a result of the test failures, the program has initiated a reliability improvement effort that includes ground and flight testing. A program official reported that the cost of reliability improvements for fiscal years 2006 and 2007 totaled $39.4 million.

Manufacturing Problems Are Often Caused by Lack of Process Controls

GAO’s past work addresses the importance of capturing manufacturing knowledge in a timely manner as a means for ensuring that an organization can produce a product within quality targets. Prime contractor activities to capture manufacturing knowledge should include identifying critical characteristics of the product’s design and then the critical manufacturing processes to achieve these characteristics. Once done, those processes should be proven to be in control prior to production. This would include making work instructions available, preventing and removing foreign object debris in the production process, and establishing criteria for workmanship. However, prime contractors’ lack of controlled manufacturing processes caused quality problems on several DOD weapon programs, including the F-22A and LPD 17 programs.

The F-22A, a fighter aircraft with air-to-ground attack capability being built by Lockheed Martin, entered production with less than 50 percent of the critical manufacturing processes in control. In 2000, citing budgetary constraints and specific hardware quality problems that demanded attention, the Air Force abandoned its efforts to get manufacturing processes in control prior to the start of production. Subsequently, the contractor experienced a scrap, rework, and repair rate of about 30 percent on early-production aircraft. The contractor also experienced major problems with the aircraft canopy. According to program officials, the aircraft uses a first-of-a-kind canopy, with an external metallic stealth layer. The contractor did not bring its manufacturing processes in control and the canopy cracked near the mounting holes. This problem was discovered in March 2000 and temporarily grounded the flight test aircraft. In addition, in 2006 a pilot was trapped in an F-22A for 5 hours when a defective activator prevented him from opening the canopy. According to the Air Force, when production began in 2001, the prime contractor should have been able to demonstrate that the F-22A could achieve almost 2 flying hours between maintenance. However, at that time, the contractor could demonstrate only about 40 minutes. Six years later, the contractor increased the flying hours to 97 minutes mean time, short of the Air Force’s current 3-hour requirement. The program now has budgeted an additional $400 million to improve the aircraft’s reliability and maintainability.

Northrop Grumman, the prime contractor for the LPD 17, the first ship of a new class of amphibious transport dock ships, delivered the ship to the Navy in 2005 with many quality problems resulting from poor manufacturing practices. For example, the program experienced problems with non-skid coating applications because the company did not keep the boat surface free from dirt and debris when applying the coating, which caused it to peel. As of late 2007, the problem was not fixed. In addition, the ship encountered problems with faulty welds on piping used in some of the ship’s hydraulic applications. According to the prime contractor, they could not verify that welds had been done properly. This problem required increased rework to correct the problems and reinspect all the welds. Had the problem not been discovered and weld failure had occurred, the crew and the ship could have been endangered. These problems, as well as many others, contributed to a 3-year delay and cost increase of $846 million in delivering the ship to the Navy. In June 2007, the Secretary of the Navy sent a letter to the Chairman of the Board of Northrop Grumman expressing his concerns about the contractor’s ability to construct and deliver ships that meet Navy quality standards and to meet agreed-to cost and schedule commitments.

Supplier Quality Problems Can Result in Higher Product Cost

Management of supplier quality is another problem area for DOD weapon systems. Supplier quality is particularly important because more that half of the cost of a weapon system can be attributed to material received by the prime contractor from its supplier base. While DOD prime contractors told us that they manage and control the quality of parts and material they receive from their suppliers with the help of performance reviews and process audits, we found supplier quality problems on seven of the weapon systems we reviewed. Two examples are the Wideband Global SATCOM and Patriot Advanced Capability-3 programs.

Boeing Integrated Defense Systems is the prime contractor for the Air Force and Army’s Wideband Global SATCOM communications satellite. Boeing Integrated Defense Systems discovered that one of its suppliers had installed certain fasteners incorrectly. As a result, 1,500 fasteners on each of the first three satellites had to be inspected or tested, and 148 fasteners on the first satellite had to be reworked. The DOD program office reported that the resulting 15-month schedule slip would add rework and workforce costs to the program and delay initial operating capability by 18 months. A prime contractor official estimated the cost to fix the problem was about $10 million.

In 2006, a supplier for the Patriot Advanced Capability-3 program, a long- range system that provides air and missile defense for ground combat forces, accepted non-conforming hardware for a component for the missile’s seeker. The seeker contractor had to re-inspect components and some were returned for rework. As a result of this and other problems involving poor workmanship and inadequate manufacturing controls, the supplier facility was shut down for 7 months, delaying delivery of about 100 missiles.

Prime Contractors’ Observations on Quality

We met with senior quality officials at the prime contractor companies we included in this review to discuss the problems we found. For the most part, they agreed with our assessment, and that the discipline with which a company implements its processes is a key contributor to quality outcomes. The officials discussed the importance of quality and how they are attempting to improve quality across their companies. This includes the use of Six Sigma, a tool for measuring defects and improving quality, as well as independent program reviews and improving design processes.

The senior quality officials also identified factors they believe affect the quality of DOD weapon systems, including insufficient attention to reliability by DOD during development and the prime contractor’s lack of understanding of weapon system requirements, including those for testing.

Leading Commercial Companies Use Disciplined Quality Management Practices

While there are similarities between the quality management practices of DOD prime contractors and leading commercial companies in our review, the discipline with which leading companies implement their practices contributes to the high quality of their products. According to company officials we contacted, reliability is a paramount concern for them because their customers demand products that work, and the companies must develop and produce high-quality products to sustain their competitive position in the marketplace. Leading commercial companies use disciplined, well-defined, and institutionalized practices for (1) systems engineering to ensure that a product’s requirements are achievable with available resources, such as technologies; (2) manufacturing to ensure that a product, once designed, can be produced consistently with high quality and low variability; and (3) supplier quality to ensure that their suppliers are capable of delivering high-quality parts. These practices, which were part of the companies’ larger product development processes, and other tools such as Six Sigma, provided an important foundation for producing quality products and continually improving performance.

Adherence to Systems Engineering Practices Leads to Clear Requirements and Reliable Designs

Several of the companies we met with discussed how they use systems engineering as a key practice for achieving quality outcomes. As part of Siemens Medical Solutions’ standard product development process, the company validates that product requirements are sufficiently clear, precise, measurable, and comprehensive. They ensure that requirements address quality, including requirements for reliability and readiness prior to making a commitment to developing and building a new product. Officials with Boeing Commercial Airplanes say they have shifted their view of quality into a more proactive approach, which includes a focus on “mistake-proofing” designs so that they can be assembled only one way. To help assess the producibility of critical parts designs, the company has also developed a tool that rates different attributes of the design, including clarity of engineering requirements, consequences of defects on performance or manufacturability, and verification complexity. Company officials say they use the tool’s ratings to modify designs to ensure that parts will be less prone to manufacturing and assembly error, and that its use has resulted in lower costs for scrap, rework, and repair and fewer quality problems.

Space Systems/Loral also relies on well-defined and disciplined processes to develop and produce satellites. Because the company’s customers expect satellites to perform for up to 15 years, product reliability is paramount and company officials say that using systems engineering to design reliability into a satellite is essential. As part of its systems engineering activities, the company performs reliability assessments to verify that satellite components and subsystems will meet reliability requirements and to identify potential hardware problems early in the design cycle. Space Systems/Loral officials also discussed testing and its importance to developing products. For significant new product developments, Space Systems/Loral employs highly accelerated life testing to find weak links in a design and correct them to make the product more robust before going into production. As a result of the company’s disciplined quality management practices, new satellite components— such as lithium-ion batteries, stationary plasma thrusters, and a satellite control system—have over 80 million hours of operation in orbit with only one component failure, according to company data.

Effective Manufacturing Process Controls Reduce Variability and Defects

Several company officials discussed the importance of having controlled manufacturing processes, and described several approaches to reduce variability and the likelihood of defects. These approaches greatly increase the likelihood that a product, once designed, can be produced consistently and with high quality and low variability. In this way, they reduce waste and increase a product’s reliability in the field.

Early in its product development process, Cummins, a manufacturer of diesel and natural gas-powered engines, establishes a capability growth plan for manufacturing processes. This increases the probability that the manufacturing process will consistently produce parts that meet specifications. Prior to beginning production, Cummins completes what it calls “alpha” and “beta” builds, which are prototypes intended to validate the product’s design and production processes. Cummins officials noted that these activities allow them to catch problems earlier in development, when problems are less costly to fix.

Officials from Kenworth, a manufacturer of heavy- and medium-duty trucks, described several initiatives it uses to improve manufacturing process controls. For example, the company has a new electronic system for process documents. Workers on the manufacturing floor used to rely on paper installation instructions, and sometimes workers used outdated instructions. Kenworth officials say that converting to an electronic system ensures that all workers use the most current process configuration and reduces rework. For a selected number of processes, Kenworth has also developed documents that include pictures as well as engineering specifications to ensure that workers follow the correct processes, and performs audits to assess whether workers are properly trained and know where to go if they have questions regarding the process.

Companies Hold Suppliers Accountable to Deliver High Quality Parts for the Product

At several of the companies we visited, officials reported that supplier parts accounted for a substantial amount of the overall product value. Companies we met with systematically manage and oversee their supply chain through such activities as regular supplier audits and performance evaluations of quality and delivery, among other things. Several officials noted that their supplier oversight focuses on first-tier suppliers, with limited interaction and oversight of lower-tier suppliers. However, Kenworth officials said they hold their first-tier suppliers accountable for quality problems attributable to lower-tier suppliers.

Leading commercial companies we met with set high expectations for supplier quality. Boeing Commercial Airplanes categorizes its suppliers by rates of defective parts per million. To achieve the highest rating level, a supplier must exhibit more than 99 percent part conformance, and company officials said they have been raising their supplier quality expectations over time. The organization has taken steps to reduce the number of direct suppliers and retain higher-performing suppliers in the supply base. Similarly, suppliers of major components for Siemens Medical Solutions’ ultrasound systems must provide conforming products 98 percent of the time, and the company will levy financial penalties against suppliers that do not meet this standard. Other companies also financially penalized suppliers for providing nonconforming parts.

Disciplined Processes and Continuous Quality Improvement Are a Focus at Several Companies

Several company officials discussed how a focus on improving product development processes and product quality served as the foundation for their systems engineering, manufacturing, and supplier quality practices. Officials with Space Systems/Loral discussed how they adopted a more disciplined product development process following quality problems in the 1990s with some of its satellites. This included creating companywide product development processes, adopting a formal program that institutionalized an iterative development process, and implementing strict documentation requirements and pass/fail criteria. The company also established an oversight organization to ensure that processes are followed. As a result, the first-year failure rate for Space Systems/Loral’s satellites decreased by approximately 50 percent from 2000 through 2006.

Likewise, Cummins officials told us that quality problems following the initial release of their ISX engine were a major factor in the implementation of their current product development process. This includes review gates to ensure process compliance and management reviews that use knowledge-based approaches for evaluating projects.

Cummins and Kenworth also use tools such as Six Sigma to define, measure, analyze, control, and continually improve their processes. For example, Cummins applies Six Sigma to its technology development, design, and production activities. The company also expects its critical suppliers to implement Six Sigma programs to improve quality and customer satisfaction. As a result of implementing initiatives such as Six Sigma, Cummins officials reported that the company’s warranty costs have declined substantially in the last several years. Kenworth also uses Six Sigma to drive efficiencies into the organization’s work processes, particularly in the design phase of new product development and in controlling manufacturing processes. Kenworth requires its first-tier suppliers to participate in a Six Sigma program. Company officials estimated that Six Sigma projects saved its Chillicothe, Ohio, facility several million dollars in 2006.

In addition, each of the commercial companies we met with collected and used data to measure and evaluate their processes and products. This helps them gauge the quality of their products and identify areas that need improvement. For example, Cummins tracks warranty costs as a measure of product quality, while Siemens Medical Solutions measures manufacturing process yields for its ultrasound systems.

Different Environments Create Different Incentives to Improve Quality

The quality problems in our case studies and the practices that relate to them—whether systems engineering, manufacturing, or supplier quality practices—are strongly influenced and often the result of larger environmental factors. DOD’s acquisition environment is not wholly conducive to incentivizing prime contractors to efficiently build high- quality weapon systems—ones that perform as expected, can be depended on to perform when needed, and are delivered on time and within cost estimates. During systems development, DOD usually pays for a contractor’s best efforts, which can include efforts to achieve overly optimistic requirements. In such an environment, seeking to achieve overly optimistic requirements along with a lack of oversight over the development process contributes to quality problems. In contrast, commercial companies we visited operate in an environment that requires their own investment of significant funds to develop new products before they are able to sell them and recoup that investment. This high-cost environment creates incentives for reasonable requirements and best practices, as well as continuous improvement in systems engineering, manufacturing, and supplier quality.

DOD’s Environment

DOD uses cost-reimbursement contracts with prime contractors for the development of its weapon systems. In this type of contract arrangement, DOD accepts most of the financial risks associated with development because of technical uncertainties. Because DOD often sets overly optimistic requirements for new weapon systems that require new and unproven technologies, development cycles can take up to 15 years. The financial risk tied to achieving these requirements during development is not borne by the contractor in this environment, but by the government. This environment provides little incentive for contractors to utilize the best systems engineering, manufacturing, and supplier quality practices discussed earlier in this report to ensure manageable requirements, stable designs, and controlled manufacturing processes to hold costs down. Finally, DOD’s quality organizations, which collect information about prime contractors’ quality systems and problems, provide limited oversight of prime contractor activities and do not aggregate quality data in a manner that helps decision makers assess or identify systemic quality problems.

Overly Optimistic Requirements Hamper Good Quality Outcomes

DOD’s ability to obtain a high-quality weapon system is adversely impacted by an environment where it both (1) assumes most of the financial risks associated with technical or cost uncertainties for the systems development and (2) sets requirements without adequate systems engineering knowledge. Without requirements that have been thoroughly analyzed for feasibility, development costs are impossible to estimate and are likely to grow out of control.

DOD typically assumes most of the financial risk associated with a new weapon system’s development by establishing cost reimbursement contracts with prime contractors. In essence, this means that prime contractors are asked to give their best effort to complete the contract and DOD pays for allowable costs, which often includes fixing quality problems experienced as part of the effort. As stated earlier, these problems can cost millions of dollars to fix. For example, DOD as the customer for the Expeditionary Fighting Vehicle signed a cost reimbursement contract with the prime contractor, General Dynamics, to develop a new weapon system that would meet performance and reliability requirements that had not yet been adequately informed by systems engineering analysis. Once General Dynamics performed a detailed requirements analysis, it informed DOD that more resources would be needed to meet the key reliability requirement established earlier. DOD decided not to invest the additional money at that time. However, when the vehicle was unable to meet its reliability goal prior to moving into production, DOD eventually decided to invest an additional $750 million into its development program to meet the reliability goal.

Often DOD enters into contracts with prime contractors before requirements for the weapon systems have been properly analyzed. For example, in March 2007 we reported that only 16 percent of the 62 DOD weapon system programs we reviewed had mature technologies to meet requirements at the start of development. The prime contractors on these programs ignored best systems engineering practices and relied on immature technologies that carry significant unknowns about whether they are ready for integration into a product. The situation is exacerbated when DOD adds or changes requirements to reflect evolving threats. Prime contractors must then spend time and resources redesigning the weapon system, flowing down the design changes to its suppliers, and developing new manufacturing plans. In some cases, special manufacturing tools the prime contractor thought it was going to use might have to be scrapped and new tooling procured.

Lack of detailed requirements analysis, for example, caused significant problems for the Advanced Threat Infrared Countermeasure/Common Missile Warning System program. Prior to 1995, the services managed portions of the program separately. Then, in 1995, DOD combined the efforts and quickly put a developer on contract. This decision resulted in significant requirements growth and presented major design and manufacturing difficulties for the prime contractor. It took over a year to determine that the tactical fixed-wing aircraft requirements were incorrect. The extent of the shortfall, however, did not become evident until the critical design review and numerous changes were required in the contract statement of work. More than 4 years after the system’s critical design review, the sensor units were built in prototype shops, with engineers only then trying to identify critical manufacturing processes. Further, sensor manufacturing was slowed by significant rework, and at one point was halted while the contractor addressed configuration control problems. The Navy and Air Force, which required the system for fixed- wing aircraft, dropped out of the program in 2000 and 2001, respectively.

Ultimately, quality is defined in large part by reliability. But, in DOD’s environment, reliability is not usually emphasized when a program begins, which forces the department to fund more costly redesign or retrofit activities when reliability problems surface later in development or after a system is fielded. The F-22A program illustrates this point. Because DOD as the customer assumed most of the financial risk on the program, it made the decision that system development resources primarily should be focused on requirements other than reliability, leading to costly quality problems. After 7 years in production, the Air Force had to budget an additional unplanned $400 million for the F-22A to address numerous quality problems and help the system achieve its baseline reliability requirements.

Oversight of Development Programs Could Be Strengthened

DOD oversight of prime contractor activities varies and has decreased as its quality assurance workforce has decreased. Weapon system progress reviews at key decision points are a primary means for DOD to oversee prime contractor performance in building high-quality systems, but they are not used consistently across programs. The purpose of the reviews is to determine if the program has demonstrated sufficient progress to advance to the next stage of product development or to enter production. The department has developed decision criteria for moving through each phase of development and production; and DOD’s acquisition executive has the authority to prevent programs from progressing to later stages of development if requisite knowledge has not been attained. Unfortunately most programs are allowed to advance without demonstrating sufficient knowledge. For example, in our recent review of 62 DOD weapon systems, we found that only 27 percent of the programs demonstrated that they had attained a stable design at the completion of the design phase.

In addition, as a result of downsizing efforts over the past 15 years, DOD’s oversight of prime contractor and major supplier manufacturing processes varies from system to system. DOD quality officials stated that they have had to scale back on the amount of oversight they can provide, focusing only on the specific areas that the weapon system program managers ask them to review. It is unclear what impact the reduction in quality assurance specialists and the reduction of oversight has had on the department’s ability to influence quality outcomes. However, in the case of the Advanced SEAL Delivery System, a lapse in effective management oversight exercised by both the government and contractor contributed to very late discovery of costly quality problems. DOD quality organizations such as the Defense Contract Management Agency do capture a significant amount of information electronically about the quality of DOD weapon systems through audits and corrective action reports. They collect quality data on a program by program basis and share information about certain types of deficiencies and nonconforming parts they found. While the organizations are looking for additional opportunities to share information, they do not currently aggregate and consolidate the information in a manner that would allow the department to determine the overall quality of products it receives from prime contractors or to identify quality related systemic problems or trends with its prime contractors.

Commercial Environment

Commercial companies must develop and deliver high-quality, highly capable products to markets on-time or suffer financial loss. The companies face competition and, therefore, their customers can choose someone else’s products when they are not satisfied. It is this environment that incentivizes manufacturers to implement and use best practices to improve quality and reduce cost while delivering on-time. Commercial customers must set achievable product requirements for their manufacturers that they know will result in a reliable, high-quality, and desirable product that can be delivered on-time. Manufacturers then get their key manufacturing processes in control to reduce inconsistencies in the product. Commercial customers understand the need to monitor and track manufacturer and supplier quality performance over time to determine which companies they want to do business with in the future or to identify problem areas that need to be corrected.

Commercial customers we visited—American Airlines and Intelsat— expect to operate their products for 30 and 15 years, respectively. The companies focus a great deal of attention on setting performance and reliability goals that manufacturers like Boeing Commercial Airplanes and Space Systems/Loral must meet in order for them to purchase their products. This provides a strong, direct incentive for manufacturers and their customers to ensure that requirements are clear and achievable with available resources, including mature technologies, before the manufacturer will invest in a product’s development. For example, Intelsat expects its satellites to be available at least 99.995 percent of the time. To meet this goal, Intelsat expects its manufacturers to use mature technologies and parts where the reliability is already known. There are several reasons that drive this approach. The most obvious one is that there is no way to fix mechanical problems once a satellite has been launched. Another reason is that the company must credit television networks, telephone companies, or cable companies for any loss of service. The company also insures their satellite for launch plus the first year of in-orbit service. Having a proven record of in-orbit performance and using reliable and flight proven technology are two important factors that help the company get favorable terms from the insurance underwriters. And, the company does not want to spend a large sum of money for a replacement satellite prior to its design life since it will negatively impact the company’s financial performance.

In the commercial environment, manufacturers are motivated to develop and provide high-quality products because their profit is tied to customer expectations and satisfaction. For example, American Airlines makes an initial payment to Boeing Commercial Airplanes when it places an order for new aircraft, but will not make final payment until it is satisfied that their requirements have been met.

In an another example, Cummins officials discussed how they were motivated to adopt more disciplined product development processes following the development effort for one of its highest selling engines, in the late 1990s. According to company officials, the design requirements were unstable from the start of development. They were changed and added to as development progressed, often without the benefit of timely and disciplined requirements analysis to ensure they could be met for the estimated investment cost. There were conflicting requirements (weight, size, performance, and fuel economy) that made development difficult. In addition, Cummins did not pay enough attention to reliability, focusing instead on weight and power considerations. As a result, development costs were higher than expected and, once the engine was sold, customers experienced less than expected. A Cummins official reported that the company found itself in an “intolerable” position with customers who were becoming increasingly dissatisfied.

This significant event, in which Cummins lost customer confidence, caused the company to examine its product development processes. The result of this examination was an improved product development process that requires a more cross functional and data-based approach to new development programs. The improvements resulted in better analysis and understanding of customer requirements leading to resource allocations before beginning new programs. Cummins invested in both customer satisfaction and the development and support of its products. This investment provided the motivation to adopt a more disciplined product development approach for the production of high-quality products for its customers.

Intelsat officials told us it makes progress payments to its manufacturers throughout development and production. However, the company holds about 10 to 20 percent of the contract value to award to the manufacturer after a satellite is successfully launched. According to company officials, the 10 to 20 percent is paid to the manufacturer over the expected life of the satellite, which is typically 15 years, when the satellite performs as expected.

The commercial companies also all capture information about their manufacturing processes and key suppliers’ quality. However, unlike DOD, they use the information when making purchasing decisions and determining how best to structure contracts to incentivize good quality outcomes. For example, in some cases Intelsat does not allow manufacturers to use certain suppliers whose parts do not meet specified reliability goals. In addition, Intelsat may include clauses in its contracts that require a manufacturer to conduct periodic inspections of particular suppliers.

DOD Efforts to Improve Acquisition Outcomes

DOD has long recognized its acquisition problems and has initiated numerous improvement efforts over the years to address them. A recent set of initiatives are highlighted by the Under Secretary of Defense for Acquisition, Technology and Logistics in DOD’s Defense Acquisition Transformation and Program Manager Empowerment and Accountability reports to Congress. Our analysis indicates that while none of the initiatives is aimed solely at improving the quality of DOD weapon systems or improving prime contractor quality practices, they could address some of the problems identified in this report, particularly the ones that improve the DOD requirements-setting process and limit requirements growth during development. A brief description of the initiatives is included below.

Concept Decision Reviews: DOD is pilot-testing a concept decision reviews program to provide a better framework for strategic investment decisions. A Concept Decision Committee composed of senior DOD officials is applying the reviews to four pilot programs— the Joint Lightweight Tactical Mobility program, the Integrated Air and Missile Defense program, the Global Strike Raid Scenario, and the Joint Rapid Scenario Generation program. A key aspect of the pilot programs is the early involvement and participation of systems engineering prior to concept decision. DOD expects this to provide decision makers better insight for setting firm requirements early, assessing technology options, considering alternative acquisition strategies, ensuring that new technology will mature in time to meet development and delivery schedules, and delivering systems with predictable performance to the warfighter.

Time-Defined Acquisition: Under the time-defined acquisition initiative, DOD plans to use such criteria as technology maturity, time to delivery, and requirement certainty to select the appropriate acquisition approach to provide a needed capability. The department envisions using a different acquisition approach, depending on whether a capability can be fielded in 2 years or less, more than 2 years to less than 4 years, or more than 4 years. In September 2006, the Under Secretary of Defense for Acquisition, Technology and Logistics stated that he anticipated the time-defined acquisition approach would facilitate better overall cost control and more effective use of total available resources.

Configuration Steering Boards: In July 2007, the Under Secretary of Defense for Acquisition, Technology and Logistics directed the establishment of Configuration Steering Boards for every current and future acquisition category I program in development. The boards, chaired by the service acquisition executive within each of the military services, are expected to review all requirements changes and significant technical configuration changes that have the potential to adversely affect program cost and schedule. Requirement changes are not to be approved unless funds are identified and schedule impacts are mitigated. However, the Under Secretary stated in his announcement of this initiative that such requirements changes would usually be rejected.

Key Performance Parameters/Key System Attributes: DOD has added new guidelines and procedures for establishing weapon system requirements in its Joint Capabilities Integration and Development System manual. The manual now requires that materiel availability be included as a key performance parameter for new weapon system development and that materiel reliability and ownership costs be included as key system attributes. Together, these requirements are aimed at ensuring weapon system sustainment considerations are fully assessed and addressed as part of the systems engineering process.

Award and Incentive Fees: DOD recently issued policy memorandums that reflect a change in policy related to the proper use of award and incentive fees. The memorandums emphasize the need to structure award fee contracts in ways that focus DOD and contractor efforts on meeting or exceeding cost, schedule, and performance requirements. The policy memorandums state that award fees should be linked to desired outcomes and payments should be commensurate to contractor performance. It also provides guidelines for how much contractors will be paid for excellent performance, satisfactory performance, and less than satisfactory performance.

While these initiatives are not directly linked together, they have the potential to help DOD implement some of the leading commercial practices we have highlighted in the past. In particular, they could help the Under Secretary of Defense for Acquisition, Technology and Logistics ensure that DOD has a better match between warfighter needs and funding at the start of weapon system development and that technology, engineering, and production knowledge is properly considered at that time. They can also help control requirements changes and requirements growth, which can adversely affect system quality during development. The initiatives are still new and, in the case of concept decision reviews, small in scope; therefore, their effectiveness may not be known for some time.

Conclusions

DOD has developed policies that address the need for setting achievable requirements, adopting commercial quality standards, using good systems engineering practices, and overseeing supplier quality. However, DOD still has difficulty acquiring high-quality weapon systems in a cost-efficient and timely manner. While many problems are caused by poor prime contractor practices related to systems engineering, manufacturing, and supplier quality, an underlying cause lies in the environment. DOD typically assumes most of the financial risk associated with development of complex systems. However, risks associated with this situation are exacerbated because DOD generally enters into development contracts without demonstrated knowledge or firm assurance that requirements are achievable, which too often result in inefficient programs and quality problems.

DOD can learn from leading commercial companies in the way they deal with risk and ensure quality in their products. Because commercial companies invest their own money in product development and recoup that investment when their customers buy the finished good, they put a new product’s requirements to the test with disciplined systems engineering practices before they commit to a large investment to develop it. If a highly valued requirement cannot be demonstrated as achievable through systems engineering, it is deferred to a subsequent product variation or to another program. Moreover, and very importantly, companies do not shortcut essential quality practices that ensure process controls and high supplier quality, including collecting and analyzing quality data. Like commercial companies, DOD must demand appropriate knowledge about requirements and make hard decisions about program risk before it initiates costly investments.

Improvements in the way DOD uses existing tools to analyze requirements during development, along with potential results of some of the initiatives it has underway, can help reduce quality risks, and address some of the long-standing acquisition problems it faces. Although the initiatives are new and in the case of the concept decision reviews, small in scope, they are a good first step toward the department setting more realistic requirements and time frames for weapon system development. Additional oversight could help ensure that prime contractors can meet requirements with given resources, such as funding and technologies, prior to DOD entering into a development contract. In addition, continued leadership from the Under Secretary of Defense for Acquisition, Technology and Logistics and a combination of actions from both DOD and prime contractors are needed to make these improvements and get the most from its planned $1.5 trillion investment in new weapons programs.

Recommendations for Executive Action

To ensure that the department is taking steps to improve the quality of weapon systems, we recommend that the Secretary of Defense take the following actions related to recent initiatives highlighted in DOD’s Defense Acquisition Transformation and Program Manager Empowerment and Accountability reports to Congress to improve its focus on setting achievable requirements and oversight:

As a part of the concept decision review initiative, have contractors perform more detailed systems engineering analysis to develop sound requirements before DOD selects a prime contractor for the systems development contract, which would help ensure that weapon system requirements, including those for reliability, are achievable with given resources.

Establish measures to gauge the success of the concept decision reviews, time-defined acquisition, and configuration steering board initiatives and properly support and expand these initiatives where appropriate.

To better assess the quality of weapon system programs and prime contractor performance, DOD needs to obtain and analyze more comprehensive data regarding prime contractors and their key suppliers. Therefore, we also recommend that the Secretary of Defense direct the Defense Contract Management Agency and the military services to: Identify and collect data that provides metrics about the effectiveness of prime contractors’ quality management system and processes by weapon system and business area over time and

Develop evaluation criteria that would allow DOD to score the performance of prime contractors’ quality management systems based on actual past performance, which could be used to improve quality and better inform DOD acquisition decision makers.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. DOD partially concurred with each of the recommendations. DOD’s comments appear in appendix III.

In its comments, DOD partially concurred with the draft recommendation that, as part of its concept decision review initiative, prime contractors should complete systems engineering analysis prior to entering a development contract. The department stated that the recommendation was vague. DOD noted that it conducts systems engineering planning prior to entering into a development contract and that prime contractors conduct more detailed systems engineering analysis afterwards. Moreover, DOD noted that systems engineering is a continuous government-performed activity at the heart of any structured development process that proceeds from concept to production. The concept decision review initiative, in particular, considers fundamental systems engineering issues such as technology and integration and manufacturing risk before the concept decision review.

To address DOD’s concern that our recommendation was too vague, we modified it to add more detail. Specifically, as part of the concept decision review initiative, we recommend that contractors that are competing for the systems development contract provide DOD more detailed systems engineering requirements analysis to be completed before a systems development contract is awarded. This would help ensure that requirements are clear and reasonable before DOD enters into a development contract. We understand that currently DOD conducts systems engineering planning prior to entering a development contract with prime contractors and that prime contractors conduct a more thorough systems engineering analysis afterwards. However, because our work has found that many DOD systems development efforts have been hampered by poorly defined or poorly understood requirements, we believe that DOD should test, through the concept decision initiative, paying contractors to complete a more thorough systems engineering analysis prior to entering into a development contract. This would give the department the benefit of more knowledge when finalizing requirements and provide an opportunity for DOD to set requirements that can be met in a well-defined time frame, which could reduce the department’s risk exposure in cost reimbursement contracts used for development. In addition, it would better position DOD to place more accountability on the winning contractor to meet the desired requirements within cost and schedule estimates.

DOD also partially concurred with the recommendation to establish measures to gauge the success of the concept decision reviews, time- defined acquisition, and configuration steering board initiatives and properly support and expand these initiatives where appropriate. In its response, DOD stated that changes to the concept decision review and time-defined acquisition initiatives are being considered and any changes would be reflected in an update to DOD Instruction 5000.2. DOD also stated that the configuration steering board initiative is being implemented consistent with its policy.

We are encouraged by the potential changes that could result from successful implementation of the concept decision reviews, time-defined acquisition, and configuration steering board initiatives. We believe that these three initiatives are aimed at addressing several of DOD’s systemic problems that impact weapon system quality and that the department should not lose sight of these initiatives. While the initiatives are new and untested in practice, acquisition history tells us that these policy changes alone will not be sufficient to change outcomes. We have found that measures to gauge success can help facilitate senior-level oversight that is needed to bring about significant change within an organization. We, therefore, believe this recommendation remains valid.

DOD partially concurred with the recommendation for the Defense Contract Management Agency and military services to identify and collect data that provides metrics about the effectiveness of prime contractors’ quality management systems and processes by weapon system and business area over time. In its response, DOD stated that the Defense Contract Management Agency is in the process of identifying and will eventually collect data that could be used to determine the effectiveness of prime contractors’ quality management systems. However, DOD stated that the added expense of capturing data by weapon system and business area does not seem warranted at this time. Further it commented that there is no need for the military services to engage in a similar effort to the Defense Contract Management Agency, since the agency is working in cooperation with the military services.

We are encouraged by the Defense Contract Management Agency’s efforts to identify and collect data on prime contractor quality management activities on a broad scale. As we noted in the report, this is a practice used by leading commercial companies we visited. During our review, the agency could only provide data on a weapon system by weapon system basis. We believe that data should be captured on both a weapon system and prime contractor basis and that the added expense of including data by weapon system is likely minimal, given that it is already being collected that way. Considering that DOD plans to invest about $1.5 trillion (in 2007 dollars) in its current portfolio of major weapon systems, we believe it would be valuable for DOD to know how the companies and business units responsible for delivering its high-quality weapon systems are performing as well as the quality associated with individual weapon systems. In addition, we believe the military services, particularly the Navy’s Supervisor of Shipbuilding organization, which is responsible for overseeing contractor activities for shipbuilding, should identify and collect similar data so that information collected is consistent and can be used for comparison purposes. We, therefore, believe this recommendation remains valid.

Finally, DOD partially concurred with the recommendation for the Defense Contract Management Agency and military services to develop evaluation criteria that would allow DOD to score the performance of prime contractors’ quality management systems based on actual past performance. DOD stated that it plans to develop evaluation criteria based on data the Defense Contract Management Agency plans to collect in the future. DOD does not think the military services need to develop a parallel effort because Defense Contract Management Agency data will be shared with the military services.

It was not our intent for the military services, the Defense Contract Management Agency, and the Navy’s Supervisor of Shipbuilding to have parallel efforts. Rather, we expected that they would work collaboratively on this effort. Moreover, not only do we believe DOD should know how well the prime contractors and their respective programs are performing as noted above, we also believe that DOD should know how well the prime contractors’ quality management systems are working. Again, this is a practice used by leading commercial companies we visited. We are encouraged that the Defense Contract Management Agency plans to develop evaluation criteria that would be used to score prime contractor quality management systems but believe the department should have a consistent methodology to be used across DOD. We, therefore, believe this recommendation remains valid.

We are sending copies of this report to the Secretary of Defense and interested congressional committees. We will also make copies available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

This report compares Department of Defense (DOD) and its large prime contractors’ quality management policies and practices with those of leading commercial companies—with a focus on improving the quality of DOD weapon systems. Specifically, we (1) determined the impact of quality problems on selected DOD weapon systems and defense contractors’ practices that contributed to the problems, (2) identified practices used by leading commercial companies that can be used to improve the quality of DOD weapon systems, (3) identified problems DOD faces in terms of improving quality, and (4) identified recent DOD initiatives that could improve quality.

To determine the impact of quality problems on selected DOD weapon systems and defense contractors’ practices that contribute to the problems, we selected and reviewed 11 DOD weapon systems with known deficiencies from each of the military services and identified the quality problems associated with each deficiency. The 11 were chosen to demonstrate the types of problems DOD weapon systems experience and to help focus our discussions with leading commercial companies on aspects of development that caused DOD major quality problems. The prime contractors in charge of developing these systems include six of DOD’s largest contractors; together, they are involved with a little over $1 trillion, or about 76 percent, of the $1.5 trillion (in 2006 dollars) DOD plans to spend on weapon systems in its current portfolio. Systems we reviewed, along with the prime contractors responsible for developing the systems, are: Advanced SEAL Delivery System, a battery-powered submarine funded by the Special Operations Command and developed by Northrop Grumman; Advanced Threat Infrared Countermeasure/Common Missile Warning System, a defense countermeasure system for protection against infrared guided missiles in flight funded primarily by the Army and developed by BAE Systems; Expeditionary Fighting Vehicle, an amphibious and armored tracked vehicle funded by the Navy for the Marine Corps and developed by General Dynamics; F-22A, an air superiority fighter with an air-to-ground attack capability funded by the Air Force and developed by Lockheed Martin; Global Hawk, a high-altitude, long endurance unmanned aircraft funded by the Air Force and developed by Northrop Grumman; Joint Air-to-Surface Standoff Missile, an air-to-surface missile funded by the Air Force and developed by Lockheed Martin; LPD 17, an amphibious transport ship funded by the Navy and developed by Northrop Grumman; MH-60S, a fleet combat support helicopter funded by the Navy and developed by Sikorsky Aircraft; Patriot Advanced Capability-3, a long-range high-to-medium altitude missile system funded by the Army and developed by Lockheed Martin; V-22, a tilt rotor, vertical/short take-off and landing aircraft funded primarily by the Navy for the Marine Corps and developed jointly by Bell Helicopter Textron and Boeing Integrated Defense Systems; and Wideband Global SATCOM, a communications satellite funded by the Air Force and developed by Boeing Integrated Defense Systems.

To evaluate each of the 11 DOD weapon systems, we examined program documentation, such as deficiency reports and corrective action reports, and held discussions with quality officials from DOD program offices, the prime contractor program office, and either the Defense Contract Management Agency or the Supervisor of Shipbuilding office where appropriate. Based on information gathered through documentation and discussions, we grouped the problems into three general categories: systems engineering, manufacturing, and supplier quality. When possible, we identified the impact that quality problems had on system cost, schedule, performance, reliability, availability, or safety. After completing our weapon systems reviews, we held meetings with senior quality leaders at selected prime contractors included in our review to discuss the quality problems we found and to obtain their views on why the problems occurred.

To identify practices used by leading commercial companies that can be used to improve the quality of DOD weapon systems, we selected and visited five companies based on several criteria: companies that make products similar to DOD weapon systems in terms of complexity; companies that have been recognized in quality management literature or by quality-related associations/research centers for their high-quality products; companies that have won quality-related awards; and/or companies that have close relationships with customers when developing and producing products. We met with these companies to discuss their product development and manufacturing processes, supplier quality activities, and the quality of selected products made by these companies. Much of the information we obtained from these companies is anecdotal, due to the proprietary nature of the data that could affect their competitive standing. Several of the companies provided data on specific products, which they agreed to let us include in this report. The companies we visited and the products we discussed include: Boeing Commercial Airplanes, a leading aerospace company and a manufacturer of commercial jetliners. We met with quality officials in Seattle, Washington, and discussed the quality practices associated with the company’s short-to-medium range 737 and extended range 777 aircraft, as well as its new 787 aircraft.

Cummins Inc., a manufacturer of diesel and natural gas-powered engines for on-highway and off-highway use. We met with quality officials at their company’s headquarters location in Columbus, Indiana, and discussed the development and quality of the ISX, a heavy-duty engine.

Kenworth Truck Company, a division of PACCAR Inc. and a leading manufacturer of heavy- and medium-duty trucks. We met with quality officials at its manufacturing plant in Chillicothe, Ohio, which was named Quality Magazine’s 2006 Large Plant of the Year, to discuss the development and quality of various large trucks.

Siemens Medical Solutions, a business area within Siemens AG, which is a global producer of numerous products, including electronics, electrical equipment, and medical devices. We met with quality officials at a company facility located in Mountain View, California, and discussed the division’s quality practices for developing and manufacturing ultrasound systems such as the Sequoia ultrasound system.

Space Systems/Loral, one of the world’s premier designers, manufacturers, and integrators of geostationary satellites and satellite systems. We met with quality officials at the company’s headquarters in Palo Alto, California, and discussed the company’s quality practices for developing satellites such as the Intelsat IX series and iPSTAR satellites.

To identify problems that DOD must overcome to improve the quality of weapon systems, we reviewed processes and tools DOD can use to influence weapon system quality. These include setting requirements, participating in key decisions during weapon system development and production, using contracts to incentivize good quality, and overseeing weapon system quality and prime contractor performance. We examined these processes and tools for the 11 weapons programs we reviewed and discussed the use of these processes and tools with acquisition and quality officials from the Office of the Secretary of Defense, military services, prime contractors, Defense Contract Management Agency, and Supervisor of Shipbuilding. We also relied on previous GAO best practices and weapon system reports to identify DOD actions that contributed to poor quality outcomes. A comprehensive list of reports we considered throughout our review can be found in the related products section at the end of this report.

We met with officials at two commercial companies that purchase products manufactured by two of the leading commercial manufacturers we included in this review. These companies included: American Airlines, the largest scheduled passenger airline in the world, which has purchased aircraft from Boeing Commercial Airplanes. We met with quality officials at a major maintenance facility located in Tulsa, Oklahoma.

Intelsat, a leading provider of fixed satellite services for telecommunications, Internet, and media broadcast companies, which purchases satellites from all major satellite manufacturers in the United States and Europe. We met with officials in space systems acquisition and planning at the company’s headquarters located in Washington, D.C.

Our discussions focused on (1) the companies’ roles in establishing requirements; (2) the types of contracts they award to manufacturers and the specificity included in the contracts in terms of quality, reliability, and penalties; and (3) the amount of oversight they exercise over their suppliers’ development and manufacturing activities.

To identify recent DOD initiatives that could improve weapon system quality, we reviewed DOD’s formal response to Sections 804 and 853 of the John Warner National Defense Authorization Act for Fiscal Year 2007. This act requires DOD to report to the congressional defense committees on acquisition reform and program management initiatives. We also met with senior defense leaders to discuss the implementation status of the acquisition reform initiatives identified in DOD’s February 2007 and September 2007 reports to the committees and relied on a previous GAO report for the implementation status of planned program management improvements.

We conducted this performance audit from September 2006 to December 2007 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Quality Problems for 11 DOD Weapon Systems

This appendix summarizes the quality problems experienced by the 11 DOD weapon systems we reviewed. The problems are categorized as systems engineering, manufacturing, and/or supplier quality problems. Most of the programs had problems in more than one of these categories. These summaries do not address all quality problems experienced on the programs; rather they emphasize major problems we discussed with officials from the military services, prime contractors, and the Defense Contract Management Agency. When possible, we include the direct impact the quality problems had on the program, the corrective actions the prime contractor or DOD took to address the problems, and the change in cost estimates and quantities from the start of program development to the present.

The cost estimates were taken from DOD Selected Acquisition Reports or were program office estimates and include DOD’s research, development, test and evaluation (RDT&E) and procurement expenditures on a particular program. We did not break out the portion of these funds that were paid to prime contractors versus the amount paid to suppliers. In addition, the change in cost estimates can be the result of a number of factors, including the amount paid to fix quality problems, a decision to procure more weapons, and increased labor rates or material prices.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

Michael Sullivan (202) 512-4841 or sullivanm@gao.gov.

Acknowledgments

Key contributors to this report were Jim Fuquay, Assistant Director; Cheryl Andrew; Lily Chin; Julie Hadley; Lauren Heft; Laura Jezewski; Andrew Redd; Charlie Shivers, and Alyssa Weir.

Related GAO Products

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-07-406SP. Washington, D.C.: March 30, 2007.

Best Practices: Stronger Practices Needed to Improve DOD Technology Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006.

Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1, 2005.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, D.C.: November 15, 2005.

Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004.

Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003.

Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept Technology Demonstration. GAO-03-52. Washington, D.C.: December 2, 2002.

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001.

Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999.

Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisitions: Improved Program Outcomes Are Possible. GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.