Defense Acquisitions: Senior Leaders Should Emphasize Key Practices to Improve Weapon System Reliability
Released: 2020-01-14T00:00:00

Letter

Background

DOD acquires new weapon systems for its warfighters through a management process known as the Defense Acquisition System. This system is implemented by two key acquisition policies: DOD Directive 5000.01, which establishes the overarching framework for the Defense Acquisition System; and DOD Instruction 5000.02, which provides detailed procedures for the operation of the Defense Acquisition System and the management of acquisition programs. These policy documents establish the guiding principles for all aspects of the DOD acquisition process. Additionally, each of the military services has its own acquisition policies which incorporate and enhance the DOD acquisition guidance. Figure 2 depicts DOD’s acquisition process beginning with Milestone A in general terms.

Several entities in the Office of the Secretary of Defense and the military departments play a role in the oversight of DOD weapon system acquisition programs, including the following:

The Under Secretary of Defense for Research and Engineering is responsible for establishing policies on and supervising all aspects of defense research and engineering, technology development, technology transition, prototyping, experimentation, and developmental testing activities and programs, including the allocation of resources for defense research and engineering. DOD’s Reliability and Maintainability Engineering lead reports to this Under Secretary.

The Under Secretary of Defense for Acquisition and Sustainment is responsible for establishing policies on and supervising all matters relating to acquisition (including (1) system design, development, and production; and (2) procurement of goods and services) and sustainment (including logistics, maintenance, and materiel readiness). This organization has certain oversight responsibilities for major defense acquisition programs throughout the acquisition process, such as collecting and distributing performance data. The Under Secretary is the Defense Acquisition Executive and serves as the milestone decision authority for certain major defense acquisition programs, meaning the Under Secretary authorizes these programs to proceed through the DOD acquisition process’s major milestones.

At the military department level, the service acquisition executive, also known as the component acquisition executive, is a civilian official within a military department who is responsible for all acquisition functions within the department and can serve as the milestone decision authority. Congress has recently devolved much of the decision making authority for major defense acquisition programs from OSD to these service acquisition executives. According to a DOD official the service acquisition executive will normally assign a relevant program manager who will then assign a chief engineer or lead systems engineer and team members with responsibility for the engineering effort of a program, including the reliability engineering effort. The following officials serve as the service acquisition executive for the military departments: the Assistant Secretary of the Air Force (Acquisition, Technology, and Logistics); the Assistant Secretary of the Army (Acquisition, Logistics and Technology); and the Assistant Secretary of the Navy (Research, Development and Acquisition) for both the Navy and the Marine Corps.

Major defense acquisition program managers, who can be either civilian or military, are tasked with developing and delivering new weapon systems while balancing factors that influence cost, schedule, and performance and ensuring that systems are high quality, supportable, reliable, and effective.

DOD’s Approach to Reliability

According to DOD guidance, reliability is the probability of an item to perform a required function under stated conditions for a specified period of time. DOD’s acquisition environment has changed over time and this has affected the way the Department addresses reliability. Until the late 1990s, DOD’s goal was to achieve good reliability by focusing on specific reliability engineering tasks during design and manufacturing, and early testing to prevent, detect, and correct design deficiencies. In the late 1990s, in response to various NDAAs, DOD implemented certain acquisition reforms, eliminating and consolidating acquisition functions, and reducing the number of personnel assigned to the remaining functions.

According to the Defense Science Board Task Force on Developmental Test & Evaluation, these reforms altered several aspects of the military acquisition process and DOD’s acquisition workforce. As a result, DOD lost experienced acquisition management and technical personnel. DOD officials stated this loss included reliability personnel who contributed to developmental testing and evaluation. DOD also canceled the Military Standard pertaining to reliability at this time. DOD officials explained that, after acquisition reform in the late 1990s, the department shifted much of the responsibility for reliability issues to contractors, and government personnel primarily focused on how systems performed during operational tests at the end of their development program.

In the mid to late 2000s, Congress and DOD took actions to increase the focus on reliability engineering during weapon system design and development. Both Congress and DOD took steps to elevate the importance of reliability, which has continued through 2019. Figure 3 depicts selected laws related to reliability and DOD reliability efforts over time.

Impacts of Poor Reliability on Warfighters

Poor reliability can negatively affect the warfighters through low operational availability; that is, the amount of time a system is available to execute its mission. For example, the MV-22 aircraft was less reliable than intended, and required many more spare parts than expected. When the Marine Corps deployed to Iraq, MV-22 maintainers had to cannibalize parts from some MV-22s to keep other MV-22s flying, and as a result, the Marine Corps had fewer aircraft available to fly missions.

Impacts of Poor Reliability on Operating and Support Costs

Reliability can significantly influence a weapon system’s operating and support costs, which we have previously reported account for approximately 70 percent of a weapon system’s total life-cycle cost. Operating and support costs are a reflection of how programs achieve operational availability for weapon systems. Programs can achieve operational availability by building highly reliable weapon systems or, if the systems are not highly reliable, supporting them with an extensive logistics system that can ensure spare parts and other support items are available when needed. DOD has previously reported that deficiencies in DOD weapon systems—such as high failure rates and an inability to make significant improvements in reliability—have historically limited program performance and increased operating and support costs.

Impacts of Poor Reliability on Commercial Companies

In the commercial world, the manufacturer carries most of the risks that would result from developing a product with poor reliability. Such risks include increased warranty expenses that decrease profits. For example, reliability personnel from Ford, Cummins, and Thermo Fisher Scientific explained that more reliable products cost their companies less because they do not have to dedicate as many resources to fixing systems that fail, which would lead to warranty claims.

In addition to increased costs, poor reliability can also negatively influence a company’s reputation. Ford representatives said that failures and product recalls are not just financial costs; recalls are highly publicized. A Thermo Fisher Scientific product manager explained that a customer’s bad experience can be shared in the media and negatively influence a company’s reputation. This may alter future buying behavior, especially in industries with relatively small customer bases in closely linked professional communities. This person shared a prior experience at a different company, where a design risk was identified during development. Instead of addressing the risk effectively, a standard cycle test was done to prove or disprove the risk. However, the test did not apply the stress necessary to cause the failure. The product was released to the market based on this successful but inadequate test. In the field, the components failed, and the company had to remove the product from the market. This damaged the company’s reputation and sales. We have previously reported that poor reliability is a concern for commercial companies because their customers demand products that work, or are reliable and do not experience failure, and the companies must develop and produce high-quality products to sustain their competitive position in the marketplace.

Commercial Companies Proactively Address Reliability

In the commercial sector, reliability engineers told us their companies proactively address reliability from the beginning of the development process. We reviewed documentation from these companies and the 2019 Reliability and Maintainability Symposium and found engineers strive to identify reliability issues at the component and sub-system level early in the development process to avoid expensive rework after producing an entire system. We identified the following key practices in the commercial sector: leveraging reliability engineers early and often, establishing realistic reliability requirements—for example, not expecting a product to operate twice as long as its predecessor before failing, emphasizing reliability with their suppliers, and employing reliability engineering activities to improve a system’s design throughout development.

Figure 4 shows some of the activities involved with these key practices.

Leverage Reliability Engineers Early and Often

We found commercial companies in our review include reliability engineers as part of their development teams. In this role, reliability engineers implement reliability tools and methods that integrate statistics, physics, and engineering principles to help develop a reliable product. For example, HBM Prenscia identified that reliability engineers from several commercial companies said it was important to initiate their assessments early in the development life cycle when there is greatest opportunity to influence product design.

According to leading reliability engineers, engineering activities can add value to decision-making by providing direction and feedback that helps development teams refine designs that lead to more reliable and cost effective systems. Researchers have reported reliability engineers should be empowered to influence decisions, such as delaying overall project schedule or negotiating for more resources when necessary. In addition, our analysis of reliability engineers’ documentation from the Reliability and Maintainability Symposium and commercial companies found it important that management provide sufficient resources and time dedicated specifically to improving reliability by discovering failures, implementing corrective actions, and verifying their effectiveness. Our analysis found that cost and schedule constraints can negatively influence reliability testing, which can limit development teams’ ability to discover failures and improve designs through corrective actions.

Our analysis of documentation from the Symposium also highlighted the importance of having experienced reliability engineers. For example, Ford representatives told us they have a dedicated reliability engineering community that coaches the members of the company’s different product development teams. Ford’s reliability engineers said they focus on teaching development team members to ask the right questions at the right point in time with the right people in the room.

Establish Realistic Reliability Requirements Based on Proven Technologies

We found companies in our review emphasize that reliability requirements should be realistic, be based on proven technologies, and reflect customer usage and the operating environment. To determine feasibility of meeting a requirement, reliability engineers we spoke with at Cummins and Thermo Fisher Scientific recommend conducting comparative analysis with historical data and assessing risk due to new, unique, or difficult technology. In addition, an independent reliability engineer with over 40 years of experience told us programs should provide justifications for how reliability requirements were established to demonstrate they are within the realm of technological possibility.

If the reliability requirement turns out not to be technically feasible, it could have broad implications for the intended mission, life-cycle costs, and other aspects of the system. We have previously reported on the importance of making informed trade-offs when considering requirements to reduce program risk or total ownership costs. HBM Prenscia representatives told us the commercial companies they work with regularly make trade-offs involving capability, reliability, and cost requirements. Reliability representatives at Ford told us it is important to have the right people involved in these trade-off decisions, and that they work with user representatives and reliability engineers to define their systems’ reliability requirements.

Emphasize Reliability with Suppliers

Systems produced by commercial companies in our review include parts or components produced by suppliers, and reliability engineers repeatedly told us the reliability of those parts or components directly impacts the reliability of the overall system. According to a leading reliability engineer, vendor quality can affect a part’s reliability, so it is critical that the reliability of vendors’ parts be evaluated before being approved for use. To emphasize reliability with suppliers, commercial companies in our review engage with suppliers early, clearly specify requirements with the supplier, and evaluate and monitor the supplier.

Cummins representatives stated engaging the supplier early is critical. They explained that they engage the supplier early, during concept development, and ask the supplier to demonstrate it can meet requirements. According to Cummins representatives, this is to ensure the supplier is able to meet quality standards and to ensure there is enough lead time and testing of components. Reliability engineers at the Reliability and Maintainability Symposium also emphasized that reliability requirements must be clearly specified with suppliers, and product teams must actively monitor suppliers and assess their deliverables. Cummins representatives explained their engineers work directly with the supplier and hold it responsible for meeting reliability requirements. Ford representatives told us they evaluate and monitor the supplier to ensure the components it is providing are reliable. For example, they visit their suppliers’ testing facilities and evaluate their testing programs, focusing specifically on their failure analysis and reliability activities. We have previously reported that leading commercial companies use disciplined quality management practices to hold suppliers accountable for high quality parts through such activities as regular supplier audits and performance evaluations.

A Thermo Fisher Scientific product manager provided a scenario where relying on an external supplier’s quality assurances would be insufficient. For example, a compressor is a critical – and commonly outsourced— component in complex industrial equipment. The product manager recommended in-house testing for critical components like a compressor rather than relying on a supplier’s testing that may not factor in real-world operating conditions. In house testing is recommended to avoid finding a failure after the product is brought to market. Post-sale failures result in dissatisfied customers, reputation damage, warranty claims and similar issues. The Thermo Fisher Scientific product manager said, in some cases, a company should establish a dedicated test facility for vital outsourced components provided by suppliers.

Employ Reliability Engineering Activities to Improve a System’s Design throughout Development

Based on our review of commercial sector practices, we found companies use reliability engineering activities to identify potential product failures and their causes. They also use these activities to improve a system’s design early and often throughout development to avoid surprises that lead to expensive rework or excessive repairs after integrating components and subsystems. For example, HBM Prenscia representatives told us that failures should be identified early, and that identification should be viewed as an opportunity to improve the design and make the product better. According to leading reliability engineers, the earlier changes are made to designs, the less costly they are to the program. It is expensive, time consuming, and risky to make changes late in development, as late changes jeopardize product reliability. The commercial company representatives we spoke with also emphasized the need to conduct reliability engineering activities iteratively until the design is optimized. For example, HBM Prenscia has identified that a common mistake is establishing a reliability plan but not actively utilizing it throughout development.

Reliability engineers use various reliability engineering activities to increase system reliability, and generally refer to these activities as design for reliability tools. These tools can be tailored to meet the specific needs of a particular development project, and can complement one another and increase reliability prior to any testing. These tools can help identify how long a part or component will work properly, how a part or component’s failure will affect a system, and what actions are needed to correct failures. See table 2 for some examples of design for reliability tools that can be used to help meet reliability goals.

We have previously reported that leading commercial companies use a knowledge-based development process that enables decision makers to be reasonably certain that product quality, reliability, and timeliness are assured. Our analysis of documentation from reliability engineers found that reliability engineering activities should be integrated into the product development process, and their outputs should be reviewed at development milestones. These reviews can help ensure that reliability is a robust process rather than a paper exercise by providing an opportunity to assess data from reliability analysis or testing. For example, Cummins incorporates reliability reviews into its product development processes to ensure products meet reliability goals prior to moving to the next phase of development. This helps ensure the company is on track to fulfill its reliability commitments and will be able to deliver the promised product reliability to customers.

The leading commercial practices we reviewed highlight the importance of consistently collecting, sharing, and analyzing data from reliability engineering activities to inform development efforts. Commercial companies we spoke with recognized the value of reliability data. For example, Cummins representatives stated they capture reliability data and share it across different product development teams to help inform estimates of reliability for new product development efforts. In addition, Cummins representatives noted that they are moving to an interactive database that personnel throughout the entire company can access. Similarly, HBM Prenscia representatives told us that failures and lessons learned from previous projects should be captured and shared within a company, and that doing so could help inform future product development efforts.

Selected Major Defense Acquisition Programs Did Not Consistently Reflect Key Reliability Practices

We reviewed seven major defense acquisition programs and found they often reactively addressed reliability after identifying issues later in development. As shown below, these programs did not consistently reflect key practices we identified in the commercial sector, and instead prioritized other activities intended to have positive acquisition cost and schedule impacts. However, DOD officials noted that there has recently been a greater emphasis on reliability, and the three programs that started development in 2012 and 2014 reflected more of the key practices than the older programs. See figure 5, which notes a distinction between commercial companies’ suppliers and DOD contractors. For more detailed information on each program, see appendix I.

Two of the Seven Selected Programs Did Not Leverage Government Reliability Engineers in Decision Making Early

The Expeditionary Fighting Vehicle (EFV) and F-22 programs did not involve reliability engineers early during system development. Instead, these programs leveraged engineers after reliability problems arose, including after they integrated components and subsystems and during system-level testing. At the end of system development, the programs brought in additional engineers and established more concerted reliability growth efforts.

In one example, the EFV program did not have an overall systems engineer. Marine Corps acquisition officials stated that reliability was not a priority during the original system development process, and we have previously reported the program was instead focused on achieving other performance parameters, including water speed, survivability, and lethality. Prime contractor representatives identified some of their design engineers who lacked experience and did not comply with engineering standards as a root cause for problems discovered late in the development process. We also reported the lack of early systems engineering discipline and knowledge undermined the EFV program’s ability to develop informed and reasonable reliability requirements, delayed the identification of potential failures until integration, and contributed to poor vehicle reliability. In addition to frequent hydraulic system failures, leaks, and pressure problems, the EFV also suffered main computer failures that froze steering while operating in water.

As we have previously reported, the EFV program was subsequently restructured. The program office hired additional engineers and consulted with Army reliability engineers to institute a reliability growth program. This program was intended to mitigate previously identified vehicle design issues related to reliability and other risks before proceeding into a second development and demonstration phase. However, the EFV program never got to fully realize the benefits of its new reliability approach, as less than 3 years after restarting development it was canceled due to continuing technology problems, development delays, and affordability concerns.

For the F-22 program, officials stated that at points during development the program did not have a leadership position focused on reliability, and the official who oversaw reliability was also responsible for supply chain management. The officials noted that at the time these were not focus areas because the Air Force expected the contractor to conduct the needed reliability engineering. In 2004, we reported that, as early as low- rate initial production, however, the Air Force identified 68 parts that had a high rate of failure and needed to be removed or replaced, requiring additional contractor work. We also reported the F-22 canopy also experienced failures during testing, allowing it to achieve only about 15 percent of its expected lifetime. In 2014, we reported that later reliability maturation projects intended to address reliability deficiencies had a positive effect on availability over time, but as of 2018 the F-22 still had not met its availability target.

Four of the Seven Selected Programs Initially Pursued Unrealistic Operational Requirements for Reliability

As we have found in our prior reports as well as in this review, the EFV, F-22, F-35, and V-22 programs set unrealistic operational requirements for reliability. These requirements were, therefore, unachievable during development and before fielding the systems to warfighters. As we have previously reported, when programs overpromise a weapon’s prospective performance and deliver systems that cannot achieve their requirements, such as reliability goals, the warfighter receives less capability than originally promised.

In one example, as we reported in 2019, more than 11 years after the start of F-35 production, none of the three aircraft variants (Air Force, Marine Corps, and Navy) had met the minimum targets for two of the program’s five reliability metrics. These include mean flight hours between part removals for replacement and mean flight hours between critical failures. We found that only the Navy variant had achieved the minimum target for a third goal, mean flight hours between maintenance events. As we reported, while the program has instituted an effort intended to improve reliability, the effort does not align improvement projects with the F-35’s reliability requirements. That is, the reliability improvement projects being funded may not improve the F-35’s performance against its reliability metrics. Ultimately, the program does not expect to achieve the unmet reliability metrics by full aircraft maturity, and program officials have acknowledged that the requirements should be reevaluated. As a result, the warfighter may not receive an aircraft that is as reliable as was expected.

In a review of the V-22 program, DOD found that the program integrated complex technologies and unprecedented capabilities into its weapon system without accounting for unknown reliability risks. Specifically, these capabilities included a conceptually new design and multiple service and mission needs. However, officials stated that the program derived its reliability requirements from antecedent helicopters, systems that were not representative of the V-22 given its increased complexity. With a limited understanding of the V-22’s mission profile, program officials stated that they also underestimated the amount of time the system would be used in helicopter mode and its operating time on the ground. Subsequently, when the Marine Corps variant of the V-22—the MV-22—was deployed in Iraq from 2007 to 2009, a number of components experienced high rates of failure, affecting systems such as the engines and engine housing. This situation, combined with an immature parts supply chain, reduced the system’s availability significantly below minimum levels. At the time, as we reported in May 2009, the MV-22 had a stated minimum mission capability rate of 82 percent, but the three MV-22 squadrons in Iraq demonstrated an average of 62 percent.

The development and integration of new technologies on the F-22— stealth, supersonics, and integrated avionics—were critical to achieving operational success, but also presented significant reliability risks. Officials told us that the F-22 was initially expected to cost less to acquire and operate than one of its predecessors, the F-15, and be more reliable as well. However, they also stated this was an unrealistic expectation. We have previously reported that the immaturity of technologies at the start of and throughout development weaken a system’s ability to achieve reliability requirements. Since 2005, when full rate production of the F- 22 began, the program has made substantial additional investments in increasing the system’s reliability through various improvement programs. But the program also changed its mean time between maintenance reliability requirement to an operational availability metric, a target that as of 2018 it had yet to meet and may need to reevaluate, according to program officials. If the F-22 cannot achieve its current reliability requirement, warfighters will have to execute their missions with a less capable aircraft than expected.

Four of the Seven Selected Programs Did Not Effectively Emphasize Reliability with Contractors

The AMPV, EFV, F-35, and V-22 programs did not effectively emphasize reliability with DOD contractors. Specifically, according to DOD, the AMPV, EFV, and V-22 did not effectively incentivize reliability with the contractor and one program, the F-35, did not include all of the program’s reliability metrics in the contract.

Each F-35 aircraft variant is measured against five reliability metrics, two of which are in part of the contract. Contractors are not responsible for achieving reliability requirements if programs do not include them in contracts. As of August 2018, two of the F-35’s three variants had not met minimum targets for any of the three metrics that are not in the contract. The last variant (Navy) has met the minimum target for only one of the three metrics. As we have previously reported, the warfighter may have to accept F-35 aircraft that are less reliable and more costly than originally expected.

As we have reported, the F-35 program tried to encourage the aircraft’s manufacturer to improve reliability through an incentive fee in sustainment contracts. These contracts, for sustainment services, included incentives for meeting aircraft availability. Reliability of parts is one of the factors that influences aircraft availability, because broken parts prevent aircraft from flying. Program officials told us they hoped the incentive fee in the sustainment contract would incentivize the contractor to invest in and implement additional reliability activities, which would help improve aircraft availability, but according to the program office, the incentive has not been effective. Program officials told us the contractor has not pursued the incentive fee in the sustainment contract through efforts to improve aircraft reliability because it would have to invest significant resources to design and incorporate changes into production aircraft in order to do so. F-35 aircraft, especially early production aircraft, continue to face challenges related to parts that are failing more often than planned and are in short supply. For example, we have previously reported that DOD found the special coating on the F-35 canopy that helps maintain the aircraft’s stealth failed more frequently than expected and that the manufacturer could not produce enough canopies to meet demand, ultimately degrading system capability.

According to program officials, to ensure that reliability growth was on track, the AMPV program offered an incentive fee of up to $16 million if the contractor could demonstrate at least 80 percent of the system’s reliability before low rate production. But officials stated that the AMPV contractor did not achieve the goal. The AMPV was a derivative system of the Army’s Bradley Fighting Vehicle with an accelerated development schedule, and officials stated that for this reason the contractor assumed the government would accept much of the Bradley’s initial design and changes to the AMPV’s performance resulting from legacy reliability issues. As a result of these expectations, officials stated that the contractor did not put enough resources, including a robust reliability team, toward the work that was eventually needed to improve reliability, and the contractor understaffed in this area.

Five of the Seven Selected Programs Deferred Key Reliability Engineering Activities until Later in Development

The AMPV, EFV, F-22, F-35, and V-22 programs deferred key reliability engineering activities, intended to improve system designs, until later in development. As a result, they missed opportunities to identify, understand, and mitigate reliability issues early in the development process. After realizing reliability shortfalls late in development, some programs initiated expensive redesign efforts that continued well into production and deployment, while others accepted degraded performance.

Based on our prior reporting, we found the EFV program did not implement a proactive reliability approach, which would include identifying challenges early and designing reliability into the system in a cost- effective manner. Instead, the program used a test-fix-test approach that relied on identifying failure modes after the system-integration phase. Early in the acquisition process, officials noted in program documentation that the program had conducted little reliability growth planning before starting development, and officials stated that the EFV program did not plan for or conduct dedicated reliability testing. Then, the program prematurely conducted its critical design review, a key review during the development phase which confirms the system’s design is stable and is expected to meet system performance requirements, before the EFV prototype’s system-integration work was complete. The program did not have the time necessary to demonstrate design maturity as scheduled and officials stated that they did not schedule long enough corrective action periods to allow for proper failure mitigation. As a result, during a 2006 operational assessment, the EFV demonstrated very low reliability and failed to complete amphibious, gunnery, and land mobility tests.

F-22 program officials stated that many of the aircraft’s components and subsystems had to be tested as part of an integrated system. This limited the discovery of reliability issues early in the development phase. DOD reliability experts told us programs should not use integrated system testing to demonstrate individual component reliability, and should instead use it to focus on how components work together and identify more complex system failure modes. F-22 officials also stated the program office frequently continued with development and other testing before implementing corrective actions for critical reliability issues. As we have previously reported, the F-22 program started a program to improve its reliability in 2005, near the start of full rate production, to mitigate hundreds of known reliability issues deferred from earlier in development. Nonetheless, we reported in 2012—nearly 3 years after DOD announced the end of F-22 production—that reliability deficiencies had increased support costs, and continued to prevent the aircraft from meeting its reliability requirement.

According to program officials, the Army selected a derivative of the Bradley Fighting Vehicle to meet the AMPV requirements, even though that vehicle’s transmission had known reliability problems. According to AMPV program officials, the Army selected this vehicle because it had prioritized controlling costs and accelerating schedule. Program officials stated that the focus on cost and schedule caused the contractor to underestimate the necessary reliability work at the start of development and led to a backlog of test incident reports and deferred corrective actions. According to 2018 program documentation we reviewed, the AMPV’s reliability growth did not track to targets during development and the vehicle did not achieve its pre-production reliability goal. Moreover, some of the AMPV’s deferred work may need to be addressed during a future corrective action period that could continue through fiscal year 2021.

DOD Acquisition Policy and Guidance Documents Identify, but Do Not Emphasize, Key Reliability Practices

Although there are differences between the DOD and commercial sector stemming from the statutory and regulatory structures that govern DOD’s acquisition processes, DOD has had long-established policy and guidance at both the department and service level that recognize the four key reliability practices we found in the commercial sector. For example, the Defense Acquisition Guidebook encourages acquisition programs to involve reliability engineers early and often, and DOD Instruction 5000.02 identifies the need for establishing realistic reliability requirements. Additionally, the 2005 DOD Guide for Achieving Reliability, Availability, and Maintainability addresses the importance of emphasizing reliability with contractors, and the service-level policies at all three military departments establish the importance of reliability engineering activities.

However, most of these documents cover a wide range of acquisition issues or many aspects of reliability engineering, and they do not specifically emphasize the four key practices we identified in our review of the commercial sector. For example, the DOD Instruction 5000.02 is an overarching policy document covering the entire acquisition life cycle at a high level, from concept development to live fire test and evaluation, and only one section provides significant detail and direction on reliability. The service level instructions and Defense Acquisition Guidebook similarly cover the entire acquisition life cycle, and reliability is one of dozens of characteristics addressed in each document. The DOD Guide for Achieving Reliability, Availability, and Maintainability is largely focused on achieving reliability, but the reliability proponents at OSD, the Army, and the Navy said the guide is not consistently used throughout DOD, noting that it was issued in 2005 and has not been updated since.

DOD policy provides decision makers flexibility to tailor regulatory activities that acquisition programs perform when developing weapon systems. The process is inherently complex, and these decision makers must balance many factors when overseeing and executing the programs. In the absence of an emphasis on the key reliability practices we identified, we found decision makers for the programs we reviewed prioritized other activities intended to have positive acquisition schedule and cost impacts. For example, AMPV program officials told us the program eliminated 7,500 miles of contractor reliability testing in order to proceed to the next development phase more quickly, believing that there would be sufficient time later to complete corrective actions.

Recently, DOD has begun employing the Middle Tier Acquisition pathway—an alternative acquisition pathway with an objective of beginning production within 6 months and completing fielding within 5 years. This emphasis may encourage decision makers to prioritize activities that promise to reduce schedule. We found that for the programs we reviewed, however, such an approach can come at the expense of other activities, such as implementing effective reliability practices.

DOD has recently taken steps that could introduce more balance when decision makers consider trade-offs between schedule and reliability. Specifically, DOD has highlighted the importance of one of the four key reliability practices we identified: emphasizing reliability with contractors, and Congress has passed legislation related to reliability. The NDAA for fiscal year 2018 included a provision mandating DOD program managers to include certain reliability requirements in weapon system engineering and manufacturing development and production contracts. In January 2019, the USD(A&S) implemented the NDAA by issuing a policy memorandum to Service Acquisition Executives and other DOD Directors echoing this key practice. However, USD(A&S) has not similarly emphasized the three other key reliability practices we identified in the commercial sector, nor have the Secretaries of the Air Force, Army, and Navy, who now have ultimate responsibility for most of DOD’s major acquisition programs. Specifically, these senior leaders have not emphasized the value of leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development.

As a result, it is less likely that acquisition programs will take the actions necessary to recognize and address potential reliability problems early in the development process. Without senior leadership emphasis on a broader range of key reliability practices, DOD runs the risk of delivering less reliable systems than promised to the warfighter and spending more than anticipated on rework and maintenance of major weapon systems. This risk is exacerbated in an environment where decision makers are striving to deliver systems in an accelerated manner.

Conclusions

The best opportunity to influence the reliability of a weapon system is early on during the design of the system. Decisions and tradeoffs made at that time can increase the weapon system’s reliability, help warfighters execute their missions, and decrease operating costs for years to come. However, these decisions and tradeoffs are not easy, as acquisition decision makers are tasked with managing competing priorities such as cost, schedule, and performance. Many of the DOD acquisition program examples in this report illustrate what can happen when reliability is not prioritized. The programs often approached reliability in a reactive manner, discovered problems late in the development process, and then tried to fix them through costly and time-consuming rework. The programs did not consistently adhere to key practices we identified in the commercial sector: reliability engineers were not leveraged early in the development process, reliability requirements were not realistic, reliability was not emphasized with contractors, and reliability engineering activities were not utilized throughout design and development.

Recent DOD actions have highlighted the importance of emphasizing reliability with contractors. DOD senior leaders can help improve reliability by highlighting the importance of the three other key reliability practices we identified in the commercial sector. In light of the current focus on accelerating the acquisition process, balancing the desire for speed with reliability considerations is critical. Given the delegation of acquisition decision authority to the military services, the Secretaries of the Air Force, Army, and Navy are in the best position to do so.

Recommendations for Executive Action

We are making a total of three recommendations: one each to the Air Force, the Army, and the Navy.

We recommend the Secretary of the Air Force issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 1)

We recommend the Secretary of the Army issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 2)

We recommend the Secretary of the Navy issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 3)

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. DOD’s written comments are reprinted in appendix II. DOD stated that the Air Force, Army, and Navy concur with our recommendations to their respective Departments. The comments also state that the Air Force and Navy plan to update their policies in response to our recommendations. As for the Army, the comments state that the Army Acquisition Executive will issue direction emphasizing the three key reliability practices and highlight an existing Army regulation focused on reliability engineering.

In addition to the responses to our recommendations, DOD’s written comments included technical comments that we addressed as appropriate. For example, we provided additional detail on an existing DOD policy, and clarified how a program engaged with a contractor.

We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Key Characteristics of Selected Major Defense Acquisition Programs’ Approach to Reliability

This appendix summarizes key characteristics of seven selected major defense acquisition programs’ approach to reliability. The four key characteristics are categorized as: did not leverage government reliability engineers in decision making initially pursued unrealistic operational requirements for reliability; did not effectively emphasize reliability with contractors; and, deferred key reliability engineering activities until later in development.

These summaries do not address all the reliability actions taken by each program; rather they focus on key characteristics we identified in our review of commercial companies and associated deficiencies. See figure 6, which notes a distinction between commercial companies’ suppliers and DOD contractors.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Nathan Tranquilli (Assistant Director), Julie A. Clark (Analyst-in-Charge), Lori Fields, Laura Greifner, Brendan K. Orino, LeAnna Parkey, Christine Pecora, Andrew N. Powell, Timothy Saunders, and Michael J. Sullivan made key contributions to this report.