F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates
Released: 2014-09-23T00:00:00

Letter

Background

The F-35 Lightning II program is a joint, multinational acquisition intended to develop and field an affordable family of next-generation strike fighter aircraft for the United States Air Force, Navy, Marine Corps, and eight international partners. There will be three variants of the F-35. (1) The conventional takeoff and landing (CTOL) variant, designated the F-35A, will be a multirole, stealthy strike aircraft replacement for the Air Force’s F-16 Falcon and the A-10 Thunderbolt II aircraft, and will complement the F-22A Raptor (see fig. 1). (2) The short takeoff and vertical landing (STOVL) variant, the F-35B, will be a multirole, stealthy strike fighter that will replace the Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier aircraft. (3) The carrier-suitable variant (CV), the F-35C, will provide the Navy a multirole, stealthy strike aircraft to complement the F/A-18 E/F Super Hornet. The Marine Corps will also field a limited number of F-35C CVs. Lockheed Martin is the aircraft contractor and Pratt & Whitney is the engine contractor.

DOD began the F-35 program in October 2001 with a highly concurrent acquisition strategy, resulting in substantial overlap between development, testing, and production. The program was restructured in 2003 and again in 2007 due to performance problems, cost growth, and schedule slips. After the program experienced a critical cost breach in 2010, the program was restructured once more, and in 2012 some near- term procurement quantities were deferred to a future date and the program’s cost estimates were increased. In 2013, DOD issued a report to Congress stating that the Marine Corps, Air Force, and Navy were planning to field initial operational capabilities (IOC) in 2015, 2016, and 2018, respectively. See figure 2 for a timeline of major events leading from the beginning of development.

The program is currently in low-rate initial production, and the contractor is assembling the next lot of 36 aircraft with a scheduled delivery by the end of 2015. As of June 2014, 78 aircraft have been fielded, are flying, and are being maintained at Eglin Air Force Base, Marine Corps Air Station Yuma, Edwards Air Force Base, Nellis Air Force Base, and Luke Air Force Base. The program had achieved about 18,250 cumulative flight hours for the fleet of aircraft; a total of 200,000 flight hours are required in order for the fleet to reach R+M maturity, as outlined in the program’s Operational Requirements Document. In March 2014, we found that the program had progressed in its developmental flight testing, but it continued to lag in testing of critical mission-systems software, delaying the delivery of expected warfighting capabilities.

By full-rate production, planned for fiscal year 2019, DOD would generally be required to establish adequate sustainment and support systems for the F-35. Per DOD guidance for all weapon systems acquisitions, these sustainment and support systems should be defined in a support concept that is incorporated into a sustainment strategy. For the F-35, this concept should comprise the necessary plans to conduct operations, maintenance, and sustainment throughout the system’s life cycle, with the F-35 Life Cycle Sustainment Plan serving as the principle document governing F-35 sustainment. According to the F-35 Operational Requirements Document, this concept must provide warfighting and peacetime capability with the lowest cost of ownership, and all variants must be able to deploy rapidly, sustain high mission reliability, and sustain a high sortie-generation rate.

In developing this strategy, the program has made affordability its top priority, and the F-35 Program Executive Officer reiterated this priority in his April 2014 testimony. In this testimony, the Program Executive Officer also identified areas in which the program is focusing on driving down sustainment costs and identifying improvements to meet long-term sustainment needs in order to “produce a mutually beneficial sustainment enterprise that operates, manages, and supports the global system with relevant metrics and incentives, while meeting warfighter-defined readiness and cost objectives.” However, the department has stated that it has a long history of starting programs that proved to be unaffordable, and the result of this practice has been costly program cancellations and dramatic reductions in inventory objectives. Recently updated DOD acquisition policy has placed more emphasis on including sustainment considerations as early as possible in the acquisition process. According to the policy, a successful program meets the sustainment performance requirements, remains affordable, and continues to seek cost reductions throughout the Operations and Support Phase that begins after the full-rate production decision.

In a 2012 memorandum, the Under Secretary of Defense for Acquisitions, Technology and Logistics (AT&L) acknowledged that certain F-35 program areas including the Autonomic Logistics Information System (ALIS), reliability growth, and sustainment costs require additional improvement.

ALIS. ALIS is the primary sustainment tool for the F-35 and is intended to predict maintenance and supply issues, automate logistics support processes, and provide decision aids to help reduce life-cycle sustainment costs and improve force readiness. ALIS is one of three major components that make up the F-35 air system, along with the aircraft and the engine, and comprises both hardware and software. The F-35 program is delivering ALIS capabilities incrementally, which include: operations, maintenance, supply chain, customer-support services, training, tech data, system security and external interfaces.

Reliability and Maintainability (R+M). R+M measures aircraft performance to determine how often the aircraft experiences failures and how much time it takes to repair those failures. R+M is monitored through a series of metrics that measure the intended performance of the aircraft in meeting its requirements by maturity at a cumulative 200,000 flight hours with at least 50,000 flight hours per variant. R+M drives sortie-generation rates and a reduced logistics footprint for the F-35, as well as informing program O&S costs, which are tied to the performance of the system at maturity.

O&S Costs. O&S consists of sustainment costs incurred from the initial system deployment through the end of system operations and includes all costs of operating, maintaining, and supporting a fielded system. The F-35 JPO develops an annual estimate for the O&S costs of maintaining and supporting the F-35 for 56 years.the estimate was $1.03 trillion; in 2012, it was $857 billion; and in 2013, it was $916 billion in then-year dollars. Additionally, CAPE conducts independent cost estimates on certain programs at different points in the acquisition cycle. In recent years, CAPE’s F-35 O&S cost estimate has been reported in DOD’s annual SAR as the official O&S cost estimate for the program. In 2013, that estimate was about $1.02 trillion in then-year dollars.

As stated earlier, recent changes to DOD’s acquisition guidance and policy have also focused on making sustainment decisions earlier in the acquisition process, by identifying the attributes of an effective performance-based logistics arrangement and requiring strategies that identify and manage technical data rights. Both of these areas will play an important role in the future of the F-35 program.

Performance Based Logistics (PBL).based logistics strategy ties objective metrics for delivered logistical system performance to incentives that will motivate the support provider (i.e., the contractor). In recent years, F-35 program managers have reaffirmed their commitment to employing a robust performance- based logistics system for the F-35 to help determine options for pursuing the most value-driven performance-based agreements with the support provider.

Technical Data Rights. Technical data for weapon systems include the details necessary to ensure the adequacy of performance, as well as instructions for operation, maintenance, and other actions needed to support weapon systems. Technical data are an important part of a weapon system program, such as the F-35. Identifying technical data needs and costs early in the acquisition process can help program managers maximize the potential for future competition and develop the system’s overall sustainment strategy.

DOD Is Developing an F-35 Sustainment Strategy, but Key Risks to Affordability and Operational Readiness Remain

DOD’s development of an F-35 sustainment strategy is an ongoing process that includes a number of plans, analyses, and decisions that must be finalized by the start of full-rate production—estimated to occur in fiscal year 2019. To date, the program’s sustainment strategy continues to evolve but may not be affordable, and DOD has not fully addressed several key risks to long-term affordability and operational readiness, that, if not mitigated, could affect the development of the strategy.

DOD Is Currently Developing an F-35 Sustainment Strategy, but It May Not Be Affordable

DOD has a number of sustainment planning documents that will make up the F-35 program’s overall sustainment strategy, including the Life Cycle Sustainment Plan, which, according to DOD policy, will provide the basis for sustainment activities. DOD also has past and ongoing studies and analyses that have the potential to affect and inform the overall sustainment strategy, including two Business Case Analyses and several Level of Repair Analyses. These planning documents and analyses will provide the basis for the overall sustainment strategy that will dictate sustainment operations when the program enters full-rate production. Furthermore, a recently launched initiative led by the F-35 JPO is working to establish a roadmap to the end of the design phase in 2018, which is intended to eventually result in a Future Support Construct—another element of sustainment planning that will inform decisions for long-term operations and support for the aircraft. The JPO plans to finalize the Future Support Construct in March 2015.

However, the current sustainment strategy that DOD is developing may not be affordable. The program has continued to experience cost overruns, and the recent SAR estimated the O&S cost to sustain the system for 56 years to be approximately $1 trillion. Additionally, according to DOD officials, including officials within the Office of the Secretary of Defense (OSD), the current sustainment strategy is not affordable. According to CAPE analysis, the combined O&S costs of several legacy aircraft—the F-15C/D, F-16C/D, AV-8B, and F-18A-D fleets—in 2010 exceeded $11 billion. Comparatively, based on CAPE’s 2013 O&S cost estimate, the annual cost to sustain the F-35 will be about $19.9 billion (in base year 2012 dollars) in 2040—the end of its steady-state years. This $8.8 billion difference represents an increase of more than 79 percent in annual O&S costs for the F-35 as compared with several legacy aircraft (see fig. 3). Moreover, the Program Executive Officer has continued to express concerns over the affordability of the program’s sustainment approach, stating that “F-35 sustainment costs remain a concern” and that affordability continues to be a top priority for the program.

In 2012, DOD established affordability targets for the program, stating that the cost per flying hour for the Air Force, Marine Corps, and Navy F- 35 variants could not exceed $35,200, $38,400, and $36,300 respectively. However, DOD officials, including some at OSD, stated that they believe that based on this threshold, the program is not affordable. To address increasing costs, DOD has taken steps focused on cost reduction including, but not limited to, establishing a Cost War Room in 2013 and a Readiness Cell in 2014. The Cost War Room is a collaborative group comprising DOD, Lockheed Martin, and Pratt & Whitney personnel, established with the purpose of reducing costs and providing support for transitioning to performance-based logistics. The mission of the Readiness Cell is to identify opportunities to enable F-35 availability (a sustainment metric) at more than 60 percent by 2015 across all three variants, thereby improving sustainment metrics that directly affect O&S costs. However, it is unclear whether these cost- saving measures can result in an affordable approach to the long-term operations and sustainment of the F-35 because the established affordability targets that must be achieved may not be representative of what the services can actually afford.

DOD’s current acquisition policy states that affordability constraints for procurement and sustainment should be determined early in program planning processes and should be used to ensure that capability requirements are prioritized and that cost trade-offs occur as early as possible and throughout the program’s life cycle. These constraints should not be based on cost estimates but rather on whether a component can afford the estimated costs of a system, and should be used to indicate whether actions must be taken to prevent exceeding the constraints. When DOD established affordability targets for the F-35 program in 2012, the methodology for determining that threshold was not informed by actual resource constraints within service budgets at the time. Specifically, DOD officials stated that the targets were determined by arbitrarily lowering CAPE’s estimated F-35 Cost per Flight Hour by 10 percent. However, without informed affordability constraints, based on military service budgets, DOD cannot be sure whether the costs savings it achieves through current efforts will lead to an affordable sustainment strategy, and DOD may miss additional areas for savings.

DOD Has Not Taken Steps to Fully Address Risks to Affordability and Operational Readiness That Could Affect the F-35 Sustainment Strategy

DOD faces the following three key risks to the long-term affordability and operational readiness of the F-35 program: (1) the performance of ALIS; (2) the R+M of the whole air system; and (3) the management of technical data rights. However, the department has not taken steps to fully address these risks, which could affect the development of the sustainment strategy.

ALIS is a system of systems that serves as the primary logistics tool to support operations, mission planning, and sustainment of the F-35. One official described ALIS as the brains of the aircraft, helping maintainers manage such things as aircraft health and diagnostics, supply-chain management, and necessary maintenance events. However, ALIS has experienced recurring problems, including user issues and schedule delays. Furthermore, DOD has stated that ALIS is one of the biggest impediments to the amount of sorties, or flights, that can be generated in support of testing and operations. The sortie-generation rate is a key performance parameter for the aircraft.

For example, the integration of ALIS capabilities—which are fielded in increments—has been repeatedly delayed. The most recent timeline for ALIS estimates that the final version of ALIS will be released for testing in 2017—7 years after the originally planned release date. Additionally, ALIS’s diagnostic system has not reached full functionality. Specifically, it has failed to meet basic requirements, including having the ability to identify faults and failures in the aircraft. That issue has resulted in time- consuming workarounds that could further affect sortie generation. Currently, DOD tracks issues with ALIS through an internal reporting mechanism that allows users in the field to identify problems with the system for potential fixes. For example, according to DOD officials, ALIS’s Anomaly and Failure Resolution System (AFRS) is intended to identify the necessary maintenance to repair the aircraft, but it is not yet mature. Maintainers at an F-35 operational training site told us that, as a result, they have had to use multiple approaches to identify the best maintenance solution. Once identified, the maintainer can submit this solution as an update for AFRS. However, this update must first be reviewed by field support on site and then sent—in the form of an action request—to the contractor for approval before it is integrated into AFRS. Maintenance officials told us that they have submitted several thousand action requests to date and have thereby created a backlog, leaving maintainers to wait multiple days for an approval.

Problems with ALIS are reported by users on an individual basis, and thus do not give DOD a complete view of ALIS’s system performance. Once an issue is reported, DOD decides whether to (1) defer the issue, forcing a workaround for users, or (2) pay for the contractor to develop a “fix.” For an overview of this process, see figure 4.

To fully understand system performance, DOD’s Systems Engineering Guide for Systems of Systems (SOS) states that it is important to have a set of metrics that allow for an assessment of the SOS performance traced to user requirements because the SOS will likely evolve based on incremental changes in individual systems—similar to ALIS’s incremental fielding. These metrics should measure the intended integrated behavior and performance of the SOS in actual operations versus the progress of the development of the SOS, allowing an assessment of SOS capabilities based on user requirements, such as functionality between interdependent subsystems. Additionally, according to the F-35 Operational Requirements Document, the F-35 must have a fully functional and effective logistics system (e.g., ALIS) aimed at ensuring operational readiness and availability and reducing overall life-cycle costs by the time the aircraft reaches maturity at 200,000 flight hours for all three variants with at least 50,000 per variant.

Department of Defense, Risk Management Guide for Acquisition (Washington, D.C.: August 2006). performance parameters, such as sortie generation, to ensure operational readiness and availability.

Reliability and Maintainability

DOD officials have continued to express concerns with the reliability and maintainability (R+M) of the aircraft. In his April 2014 testimony, the Program Executive Officer stated that R+M remains an area for needed improvement. In that same month, the Director, Operational Test and Evaluation (DOT&E), stated that the growth rates and improvement programs that would be necessary to meet R+M requirements when the aircraft reaches maturity (200,000 flight hours) are what they termed “ambitious.” Additionally, DOT&E stated that future efforts to improve R+M may be hampered by higher failure rates as the aircraft begins to fly with more aggressive maneuvering and more extensive use of mission systems.

To measure the R+M of the aircraft, testers collect data on (1) the number of flight hours achieved before a failure occurs (reliability) and (2) the amount of time it takes to repair those failures (maintainability). These two primary measures are supported by various metrics and tracked along planned growth curves to measure progress in meeting requirements. Based on recent growth curves and reports for some of DOD’s R+M metrics, some metrics have progressed, some continued to lag, and some have worsened. For example:

Mean Flight Hours between Failures (Design Controllable) is the average amount of flight hours achieved before a design-controllable failure occurs. As of March 2014, this metric was progressing in that the number of flight hours before a failure occurs was increasing for all three variants. For example, the average flight hours between failures for the F-35A—the variant with the most flight hours to date— was 5.2 in March 2014, surpassing the expectation at its current flight hours by about 1.2 and growing toward its requirement at maturity of 6.0. Moreover, this was an increase of about 1.8 average flight hours between failures since September 2013, as reported by GAO.

Mean Flight Hours between Critical Failures is the average amount of flight hours achieved before a failure occurs that results in the loss As of March of a capability to perform a mission-essential function.2014, this metric was lagging well below its requirements at maturity, meeting an average of 42 percent of those requirements across all three variants.

Mean Time to Repair is the average time it takes a maintainer to repair a failed component or device. Currently, this metric is not improving in that as flight hours increase, it is taking maintainers longer to repair failed components for the F-35A and F-35C, and the amount of time it takes to repair failed components for the F-35B remains unchanged. Specifically, GAO reviewed R+M growth curves provided by DOD showing the historical growth of this metric from 2009 for the F-35B and 2010 for the F-35A and F-35C to March 2014, and we observed that the metric is trending in the opposite direction of its predicted path for the F-35A and F-35C, and the metric is remaining steady, without improvement, for the F-35B.

A tenet of F-35 sustainment is to continually improve R+M of the weapon system to drive down O&S costs. In an effort to improve R+M, DOD began a Reliability Improvement Plan in 2014 that focused on addressing the top 20 issues with the aircraft that affect R+M. While the current Reliability Improvement Plan is in the process of addressing some R+M issues, the plan focuses primarily on the aircraft’s hardware and does not specifically address software and its potential effects on overall R+M. The F-35 is the most software-intensive fighter aircraft DOD has procured to date. According to DOD officials, software reliability is an issue that needs attention; however, officials have also stated that the current process for tracking R+M is almost exclusively dedicated to hardware.

To identify some software issues as they arise, users in the field use an internal system to submit requests to the contractor, but these requests are submitted on an individual basis and may not always be addressed immediately as it takes time to determine whether the issue is related to hardware or software. For example, officials told us that the Electrical Optical Targeting System, which is used to track a target, continues to fail. In this instance, testers reported the problem, and officials attempted to improve the capability with hardware changes. However, not all issues with the Electrical Optical Targeting System were fixed with the hardware changes, and officials have decided to also try to address the issue with software changes, causing users to identify workarounds in the meantime. As another example, officials discussed instances in which the diagnostics system signals to a maintainer that the landing gear failed, but it was actually a sensor near the landing gear that failed. Because software for isolating these types of failures is not yet mature, operators and maintainers on the ground may continue to check the landing gear without discovering the sensor issue and reporting it. Finally, officials stated that with the release of the next increment of software in 2015, a number of new issues may arise, and these issues may be related to software because the new software includes new processors that can affect mission systems on the aircraft.

According to DOD reliability guidance, basic reliability activities should include software reliability assessments. Additionally, Naval Air Systems Command guidance advises that weapon systems programs should establish software R+M metrics as a key activity in terms of achieving supportability goals and developing a high level of confidence that the product will work reliably and can be maintained easily. Furthermore, DOD’s Risk Management Guide states that effective risk management depends, in part, on continuous monitoring and reassessment to identify and address root causes of program risks. Currently there are no processes or metrics by which the program can monitor and assess software R+M and the full extent to which software issues may be contributing to the overall R+M issues of the aircraft. Without an assessment process, including metrics, for software R+M as a means of mitigating ongoing risks to R+M, program officials cannot have a full understanding of the root causes of R+M issues on the aircraft, or whether software failures contribute to the overall R+M outside of hardware failures. This can potentially lead to unforeseen costs in the future as software is continually upgraded and refined. Since O&S costs are based on meeting the required reliability at maturity (200,000 flight hours), there are increasing risks to O&S cost and future aircraft availability.

Technical Data

In May 2011, we found that DOD needs access to technical data— recorded information used to produce, support, maintain, or operate a system—related to its weapon systems in order to help control costs and maintain flexibility in the acquisition and sustainment of those weapon systems. Technical data can enable the government to complete maintenance work in-house, as well as to competitively award acquisition and sustainment contracts. Additionally, we found that for service contracts pertaining to DOD weapon programs, which can involve products as well as support services, the lack of access to proprietary technical data and a heavy reliance on specific contractors for expertise creates limitations to, or even precludes, the possibility of competition.

Within the past 5 years, acquisition reforms such as the Weapon Systems Acquisition Reform Act of 2009 and other initiatives seeking greater efficiency and cost savings in acquisitions have put greater emphasis on obtaining technical-data rights and on maintaining competition throughout the life cycle of weapon systems. In 2010, DOD announced its “Better Buying Power” initiative that outlines a series of actions and directives to promote competition, including ensuring that technical data requirements are considered at key milestones of the acquisition cycle. Specifically, in November 2010, the Under Secretary of Defense for Acquisition, Technology and Logistics issued a memorandum that requires program officials to (1) conduct a business case analysis that outlines the technical data rights the government will pursue to ensure competition, and (2) include the results of this analysis in certain strategies at Milestone B.

However, according to DOD officials, because the F-35 program’s initial Milestone B decision occurred back in 2001, and these initiatives started nearly a decade later, the F-35 program is getting a late start on its assessment of its technical data needs.

In accordance with the Acquisition Strategy for the October 2001 Milestone B, the F-35 program did not acquire technical data suitable for competition for the F-35 or its subsystems under its competitively awarded system development contract. Furthermore, as stated in the Acquisition Strategy, since the system design contract is in place, the acquisition of data suitable for competition may be prohibitively expensive and would not result in a commensurate return on investment. The current iteration of the Acquisition Strategy leaves the door open for acquiring current and future technical data that could support future competition through the life cycle. Specifically, DOD plans to assess the merits of acquiring operations, maintenance, installation, and training data that would support sustainment and operation efforts. Planning for follow- on procurements is to include data rights/management strategies that will support, to the extent practical, the options for competition through the remainder of the program life cycle. With respect to this assessment, and DOD’s current requirements related to technical data, DOD officials confirmed the need for the F-35 program to begin identifying technical data needs and costs related to future sustainment of the aircraft. They said that DOD has begun discussions with contractor officials to determine who, pursuant to the applicable contract clauses, has the rights to specific technical data associated with the various hardware and software systems. According to JPO officials, technical-data rights could play a significant role in the development of the sustainment strategy for the F-35; however, the federal government does not have an understanding of the technical data it currently owns, what technical data it may still need, and how much it may cost to acquire those data to support the future sustainment of the F-35.

DOD, Defense Federal Acquisition Regulation Supplement (DFARS) Subpart 207.106 (S-70). The DFARS provides DOD implementation and supplementation of the Federal Acquisition Regulation (FAR). The DFARS contains requirements of law, DOD-wide policies, delegations of FAR authorities, deviations from FAR requirements, and policies and procedures that have a significant effect beyond the internal operating procedures of DOD or a significant cost or administrative effect on contractors. DFARS § 201.301. assess the long-term technical data and computer software needs of those systems and subsystems; and establish acquisition strategies that provide for the technical data and computer software deliverables and associated license rights needed to sustain those systems and subsystems over their life cycle; the strategy may include the development of maintenance capabilities within DOD; or competition for contracts for sustainment of the systems or subsystems.

According to the current F-35 acquisition strategy, the F-35 program will assess the merits of acquiring current and future technical data that could support future competition throughout the life cycle, and will help provide the basis for identifying the data and data rights required for the sustainment of the program. However, as of July 2014, DOD officials had not completed assessments of technical data rights, including a plan that would identify what technical data rights the federal government currently owns, what additional technical data rights it may need, and the costs associated with purchasing any necessary technical data. According to DOD officials, the program is planning to address these technical data rights issues as part of the future sustainment strategy that will document decisions for long-term operations and support for the aircraft. To date, although DOD officials have acknowledged the importance of technical data rights to the future of the F-35 program and have told us that discussions with contractor officials regarding ownership of those rights have begun, we have not seen any documentation confirming that assessments of technical data rights will be completed. Until DOD determines the technical data rights the federal government currently owns and its critical technical data needs and associated costs, the program runs the risk of being limited in its flexibility to make changes to sustainment plans and the overarching sustainment strategy. Furthermore, without a long-term IP Strategy, the program runs the risk of not being able to compete requirements for such things as spare parts. Unless technical data rights needs are considered up front, critical data and software may not be acquired, rendering them unavailable (or unaffordable) years later when seeking to maximize competition on a program during its sustainment phase.

Improved Assumptions and Additional Analyses Could Increase Reliability of F-35 O&S Costs

DOD’s most recent O&S cost estimates are comprehensive in that they include all DOD-required program elements and are organized according to a standard O&S cost-estimating structure; but weaknesses exist with respect to a few of the assumptions, and the estimates did not include all analyses necessary to make them fully reliable.

DOD Created Two Primary Estimates of Operating and Support Costs for the F-35 across Its Life Cycle

There are two primary DOD F-35 O&S cost estimates, one developed by the JPO and one by CAPE. O&S costs include the direct and indirect costs of sustaining a fielded system, such as costs for replacement parts, fuel, maintenance, personnel, support facilities, and equipment (see table 1). The JPO’s 2013 estimate, which is used to drive budget planning, lists the total F-35 O&S costs across a 56-year life cycle as approximately $916 billion in then-year dollars. CAPE’s estimate, which is used to assess affordability and is required at major milestones or by special request, lists the same cost to be about $1.02 trillion in then-year dollars. Generally, either the JPO’s or CAPE’s estimate is used as DOD’s estimate of record and published in the Selected Acquisition Report (SAR)—a report that DOD is required to submit annually to Congress on each of its major defense acquisition programs.

DOD F-35 Cost Estimates Are Comprehensive, but Weaknesses Exist in a Few Assumptions

The JPO and CAPE both created comprehensive F-35 O&S cost estimates, but there are weaknesses in a few of the assumptions used in the estimates. According to the GAO Cost Estimating and Assessment Guide, a good cost estimate depends on how well the program is defined in the technical baseline and work-breakdown structure. Both the JPO and CAPE used a DOD-approved technical baseline and a product- oriented O&S cost work-breakdown structure to develop their 2013 estimates. The technical baseline provides the comprehensive program description, which is used to determine what must be included in the estimate. A work-breakdown structure is a necessary program- management tool because it provides a basic framework for estimating costs, allowing for the estimates to be easily updated and compared to future estimates. Both DOD guidance and the GAO Cost Estimating and Assessment Guide highlight the importance of good documentation in cost estimating, stating that a complete cost estimate should be formally documented to serve as an audit trail of source data, methods, and results. In addition, the GAO Cost Estimating and Assessment Guide states that all assumptions and exclusions the estimate is based on should be reasonable, clearly identified, and explained. While the JPO and CAPE estimates are comprehensive, weaknesses exist because both estimates used unreasonable assumptions for fuel burn rates and have not clearly documented intermediate-level maintenance costs (see app. II for more information on our analysis).

Fuel burn rates: The JPO and CAPE estimates do not use a reasonable assumption for the Marine Corps fuel burn rate across the life cycle of the aircraft. Both estimates use a fuel burn assumption of 1,493 gallons per flying hour for the Marine Corps variant. While the rate used in the estimates was calculated based on service-planned missions at maturity and DOD-validated physics-based models, this burn rate is an almost exact match of the burn rate being observed at the operational sites. This may be a reasonable assumption for the short term, but it is likely that the fuel burn rates will increase in the long term. The burn rate of 1,493 gallons per flying hour used in the JPO and CAPE cost estimates reflects fuel burn data from aircraft flown using limited capabilities—at slower speeds and at lower altitudes than the F-35 will eventually fly. After 2015, the Marine Corps is planning on using the aircraft’s increased capability, which will likely result in more fuel being used. In addition, shortly after 2015, the Marine Corps is planning on deploying the F-35 to ships, where the aircraft’s fuel-intensive vertical landing capability will be used more frequently, which will likely increase fuel usage. Consequently, the use of lower fuel burn rate across the entire life cycle of the Marine Corps variant is not a reasonable assumption.

Similarly, the JPO estimate lowered its fuel burn rate assumption to 1,480 gallons per flight hour from 1,558 gallons per flight hour for the Air Force variant. However, the Air Force stated that the more conservative assumption of 1,558 gallons per flight hour should be used across the life cycle because the F-35 has yet to use its full flight capabilities, weapons, or mission systems, which will likely increase the fuel burn rates in the long term. The higher fuel burn rate assumption for the Air Force would represent a $4.0 billion cost increase in base year 2012 dollars across the life cycle of the aircraft.

Intermediate-level maintenance. These costs are not clearly documented in either the JPO or the CAPE cost estimates. For the F- 35, intermediate-level maintenance tasks include wheel and tire servicing, battery maintenance, structural airframe support, and inspections. The documentation supporting DOD’s cost estimate states that intermediate-level maintenance costs are not calculated for the F-35, but the program’s maintenance planning documents state that intermediate-level maintenance will be used in the sustainment of the F-35, and it is currently being employed. Specifically, Marine Corps officials told us that intermediate-level personnel at Yuma are conducting some F-35 activities, including tire, wheel, and battery servicing. In addition to the existing F-35 intermediate-level maintenance being conducted at Yuma, DOD has conducted at least five analyses to date, each resulting in a recommendation to repair a F-35 Marine Corps variant component at the intermediate level. These five component repairs represent more than $30 million in intermediate-level maintenance costs across the life cycle of the aircraft, but they are not documented in DOD’s cost estimates. Moreover, all of the analyses to identify which components will be repaired at the intermediate level are not yet complete, and the costs related to intermediate-level maintenance could increase. According to DOD officials, the costs of these intermediate-level repairs are already included under mission personnel and unit-level maintenance elements of the cost estimates. However, this is not documented in the mission personnel or unit-level maintenance sections of the cost estimates. While DOD guidance states that intermediate-level costs can be combined with unit-level costs, the guidance also states that this should be noted in the cost estimate’s supporting documentation to avoid an interpretation that a portion of the maintenance costs were omitted. Without clear documentation of these costs it is unclear whether they are fully represented in the JPO and CAPE estimate.

In addition, the JPO estimate does not include reasonable assumptions about part-replacement rates and depot maintenance.

Part replacement: The JPO estimate does not include reasonable assumptions for part replacement. Based on data from the Air Force and Marine Corps F-35 variants at testing and operational sites, parts are being replaced, on average, 15 to 16 times more frequently than the assumptions used across the life cycle of the JPO estimate (see table 2). For example, a sensor that costs about $4,800 is being replaced 60 to 129 times more frequently than anticipated across the life cycle of the JPO cost estimate. Another example is the battery charger unit, which costs about $60,000 to acquire new, and is being replaced 3 to 8 times more frequently than anticipated across the life cycle of the JPO cost estimate.

The part-replacement assumptions used by the JPO reflect the anticipated reliability of the aircraft at maturity—once the entire fleet has achieved 200,000 flight hours. According to JPO officials, the reliability issues causing the high part-replacement rates will be resolved once the aircraft reaches maturity, which is estimated to occur at the end of fiscal year 2019. The JPO increased the cost of replacing parts in the 2010 to 2019 portion of its estimate to reflect the lower reliability of the aircraft until maturity. However, according to officials from the Institute for Defense Analysis, who conducted a study of the F-35’s R+M for DOT&E, the F-35 program would have to achieve a higher reliability-growth improvement rate than has been observed in almost all other aircraft in order to meet the anticipated reliability by 2020. As previously stated, reliability improvement efforts are under way that could reduce these costs, but it is unlikely that these efforts will bring significant results in the near term because the current F-35 fleet must be modified into the configuration necessary for reliability improvement. As a result, for the next 10 years, the F-35 fleet will not represent the configurations necessary for reliability-growth improvement. In addition, according to the study of the F-35’s R+M conducted by the Institute for Defense Analyses for DOT&E, future efforts to increase reliability may be hampered by increased failure rates as the fielded fleet starts flying more operationally representative missions, with more aggressive maneuvering and more extensive use of mission systems.

CAPE adjusted its part-replacement assumptions for its 2013 estimate to reflect higher replacement rates across the life cycle of the aircraft.

As a result, CAPE’s 2013 estimated cost for parts across the life cycle of the program is $23 billion greater than the JPO’s in base year 2012 dollars. CAPE officials told us that if they had used the actual replacement rates being observed at the F-35 sites, the part- replacement cost across the life cycle of the program would have been roughly $120 billion greater than the JPO’s in base year 2012 dollars.

Depot maintenance: Finally, the JPO estimate does not include a reasonable assumption regarding F-35 depot maintenance. The depot maintenance cost element includes the labor, materiel, and overhead costs for overhaul or rework of aircraft returned to a government or contractor repair facility. This O&S cost includes scheduled and unscheduled depot maintenance, as well as inspections. We have found this to be a cost element that has been underestimated in the past for legacy and fifth-generation aircraft, including the F/A-18 and F-22. The JPO reduced its assumptions related to the cost of depot maintenance by about half, from about $2 million per scheduled depot induction in the 2012 estimate to about $600,000–$1.3 million in the 2013 estimate. This change resulted in a $19.7 billion reduction in then-year dollars from the JPO’s 2012 estimate. JPO officials stated that this reduction was the result of using a different cost-estimation method, which had the effect of lowering the hourly cost per induction. According to the assumptions in the JPO’s 2013 estimate, the average cost per depot induction hour is roughly $205. However, in 2004, the average cost per depot induction was about $238, and it is likely that labor and materiel costs have increased over the past 10 years. To develop its hourly cost, the JPO used F-16 data from Lockheed Martin contractors that had been adjusted for the F-35. According to the JPO’s current assumptions, materiel costs would be 30 percent of the labor costs, but data for the AV-8 and F/A-18 depot inductions used in the previous year’s estimate indicate that depot induction materiel costs have historically been closer to 45 percent of labor costs. As a result, it is likely that the depot maintenance hourly cost used in the JPO’s 2013 estimate is not a reasonable assumption.

DOD Has Not Conducted Uncertainty Analyses to Determine Credibility of Estimates

Every cost estimate contains a degree of uncertainty because of the many assumptions that must be made about the future. To mitigate this uncertainty, a variety of checks and analyses can be conducted to determine the credibility of the assumptions and the estimate as a whole. While the JPO and CAPE have done some cross-checks and sensitivity analyses to mitigate the uncertainty of their assumptions, they have not conducted uncertainty analyses to determine whether the estimates reflect the most likely costs.

According to the GAO Cost Estimate and Assessment Guide and DOD guidance on cost estimating, a good practice for determining the reasonableness of an estimate’s assumptions for certain cost elements is to cross-check them with other programs’ estimates or other cost- estimate methodologies to determine whether results are similar. JPO and CAPE cost estimators stated that they cross-checked their results of key cost elements against O&S estimates for other aircraft to determine whether they produced similar results. For example, according to a JPO cost-estimating official, the F-35 results for the depot maintenance cost element were three to five times greater than the same costs for legacy aircraft. In addition, CAPE conducted a cross-check comparing the cost per flying hour of legacy and F-35 aircraft, which identified F-35 costs as being significantly higher than the legacy aircraft. The official stated that this was reasonable given the complexity of the fifth-generation aircraft and the additional capability that the F-35 offers.

The JPO and CAPE have also conducted some sensitivity analyses, but they have not conducted uncertainty analyses to determine the credibility of their estimates. Sensitivity analyses examine how changes to individual assumptions and inputs affect the estimate as a whole. Both the JPO and CAPE cost estimators have told us that they conducted “what if” scenarios—varying the assumptions of certain variables individually to see how the total O&S costs may be affected. Although CAPE did not conduct a sensitivity analysis on its 2013 O&S cost estimate, it did conduct one for its 2011 F-35 O&S cost estimate, which contains many of the same assumptions as its 2013 estimate. However, as a best practice, a sensitivity analysis should be included in all cost estimates because it examines the effects of changing assumptions and ground rules. The JPO asked officials at the Office of the Secretary of Defense, Logistics and Materiel Readiness (OSD-LMR), to conduct a sensitivity analysis using the JPO cost model. OSD-LMR conducted this analysis, determining the cost elements that represented the highest percentage of cost and varying assumptions to determine which factors have the greatest effect on the total O&S cost. According to DOD guidance, sensitivity analyses are useful for identifying critical estimating assumptions, but they have limited utility in providing a comprehensive sense of overall uncertainty. In contrast, an uncertainty or quantitative risk analysis can provide a broad overall assessment of variability in the cost estimate. Similarly, the GAO Cost Estimating and Assessment Guide states that risk and uncertainty analyses should be performed to determine the level of risk associated with the estimate. JPO and CAPE cost estimators have not conducted uncertainty analyses to understand the potential range of costs around their point estimates and to convey their level of confidence that the estimates reflect the most likely costs. JPO and CAPE officials stated that they did not conduct uncertainty analyses because they are not a standard practice for DOD when creating O&S cost estimates. However, according to the GAO Cost Estimating and Assessment Guide, an estimate without a risk and uncertainty analysis is unrealistic because it does not assess the variability in the estimate.

In April 2014 testimony, the F-35 program executive officer highlighted the importance of refining the cost estimate as a key component of DOD’s risk-reduction efforts related to the F-35. Without fully reliable O&S cost information to develop accurate budget requests and address the affordability of the F-35 program, the military services may be at risk of sustainment cost overruns in the near and long terms. The use of improved assumptions, as well as an understanding of the level of confidence in the estimates through the use of an uncertainty analysis, would also improve DOD’s ability to make F-35 sustainment decisions.

Conclusions

The F-35 program remains one of the top priorities for the future of our national defense, according to senior defense leadership. However, as we have previously reported, the F-35 program has experienced cost, schedule, and performance problems that have put it 7 years behind schedule and significantly over its original budget. With life-cycle sustainment costs estimated by DOD at about $1 trillion, the F-35 is approaching a critical juncture. In an era of significant budgetary pressures and competition for resources, with the Air Force, Navy, and Marine Corps each beginning to deploy and maintain F-35 aircraft within the next 4 years, and with full-rate production planned in fiscal year 2019, DOD must find ways to make the sustainment of the F-35 program affordable. DOD has begun to devote attention to sustainment through the creation of plans and analyses that could help inform programmatic decisions and shape its overall sustainment strategy. One key decision is for DOD to determine what it can realistically afford with respect to the F- 35 program, but until it identifies affordability constraints tied to the military services’ budgets, it will continue to develop and field the most costly weapon system program in history without knowing whether the Air Force, Navy, or Marine Corps can pay for it. Furthermore, while DOD has begun testing and fielding the F-35 at sites around the country, the program faces several risks, including ALIS, software R+M, and uncertainties related to technical data rights, which could adversely affect DOD’s sustainment strategy. If these risks remain unaddressed, sustainment costs could potentially increase well beyond current estimated levels, and operational readiness could suffer.

Finally, although DOD’s current cost estimates identify most sustainment costs and document cost-influencing assumptions, certain elements may be underestimated. In some areas, such as the assumptions underlying future fuel burn rates, continuing to underestimate these elements could translate to billions of dollars in cost increases over the life cycle of the program. Therefore, it is imperative that DOD provide an accurate picture of the current and future state of the program, as well as a greater degree of assurance that its cost estimates are credible—through the use of uncertainty analyses—to help inform the sustainment strategy. With more than 2,400 F-35 aircraft still planned for purchase, DOD has an opportunity to chart a more informed, analytical, and affordable path for the program. However, sustainment traditionally represents about 70 percent of a weapon system’s life-cycle cost, and until DOD develops a viable sustainment strategy, addresses key risks, and improves its cost estimates, the department will likely continue to present Congress with near-term funding requests without having full knowledge of the extent of the program’s long-term financial requirements.

Recommendations for Executive Action

We are making the following eight recommendations to improve DOD’s sustainment planning and operating and support (O&S) cost estimates for the F-35 program.

To help DOD develop an affordable sustainment strategy for the F-35, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisitions, Technology and Logistics to direct the F-35 Program Executive Officer to establish affordability constraints linked to, and informed by, military service budgets that will help guide sustainment decisions, prioritize requirements, and identify additional areas for savings by March 2015, at which point the Future Support Construct decision will be approved.

To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, we recommend that the Secretary of Defense direct the F-35 Program Executive Officer to take the following five actions:

To enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

To develop a high level of confidence that the aircraft will achieve its R+M goals, develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F- 35 R+M issues.

To promote competition, address affordability, and inform its overarching sustainment strategy, develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of current levels of technical data rights ownership by the federal government and all critical technical data needs and their associated costs.

To improve the reliability of the JPO F-35 O&S cost estimate, clearly document assumptions related to intermediate-level maintenance costs and revise assumptions related to fuel burn rates, part replacement, and depot-maintenance induction in its future F-35 O&S cost estimates to better reflect the current and future state of the F-35 program.

To understand the potential range of costs associated with the JPO F- 35 O&S cost estimate, conduct uncertainty analyses on future JPO estimates.

To improve the reliability of the Cost Assessment and Program Evaluation (CAPE) F-35 O&S cost estimate, we recommend that the Secretary of Defense direct the Director of CAPE to take the following two actions for future F-35 O&S cost estimates: clearly document assumptions related to intermediate-level maintenance and revise fuel burn assumptions to better reflect the current and future state of the F-35 program and conduct uncertainty analyses to understand the potential range of costs associated with its estimates to reflect the most likely costs associated with the program.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with seven of the report’s eight recommendations and partially concurred with one recommendation. DOD’s comments are summarized below and reprinted in appendix III. DOD also provided technical comments, which we have incorporated into our report where appropriate.

DOD concurred with the recommendation that the Secretary of Defense direct the Under Secretary of Defense for Acquisitions, Technology, and Logistics to direct the F-35 Program Executive Officer to establish affordability constraints linked to, and informed by, military service budgets. DOD stated that it established F-35 affordability targets for unit recurring flyaway cost and for sustainment cost in March 2012 to guide development of an affordable strategy. DOD further stated that the F-35 program is currently operating under these affordability targets and using affordability goals to reduce life-cycle costs. We acknowledge that the department established affordability targets for sustainment in March 2012, but these actions do not fully address the intent of our recommendation because, as we note in our report, these affordability targets may not be representative of what the services can actually afford because the methodology for determining the targets was not informed by resource constraints within military service budgets. We further acknowledge in our report that DOD is currently engaged in cost- reduction efforts, but until DOD establishes affordability targets that are informed by military service budgets, the department cannot be sure whether the cost savings it achieves through current efforts will lead to an affordable sustainment strategy. If DOD develops affordability targets that are linked to and driven by military service budgets, this action should address the recommendation.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to establish a performance- measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements. DOD stated that the F-35 program will continue to use ALIS performance metrics and targets to assess effects on the operational requirements, such as the sortie-generation rate key performance parameter, through the remainder of system development and demonstration (SDD). DOD also stated that while the F-35 program continues to assess these effects today, future incremental ALIS functional deliveries and additional operational experience will enable more-refined metrics over time which will help reduce performance risk and ensure a successful operational test and evaluation at the end of SDD. As our report states, we acknowledge that metrics for ALIS’s availability are used to track the development of ALIS, but these metrics do not fully address the intent of our recommendation because there is no performance-measurement process that directly ties ALIS performance to user requirements. While the sortie-generation rate key performance parameter is one way in which to monitor performance, this metric is aimed at measuring the performance of the entire F-35 air system and not, specifically, how ALIS performs its functions in response to user requirements. We agree that, as DOD noted, performance metrics and targets may be refined over time to help reduce performance risk, but until DOD develops performance targets and metrics tied directly to ALIS functionality and user requirements, it may still face additional costs and risks to operational readiness and availability. If DOD develops these performance targets and metrics as part of a performance-measurement process, this action should address our recommendation.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to develop a software reliability and maintainability assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 reliability and maintainability (R+M) issues. DOD stated that the F-35 program has an established process for assessing software reliability and will continue to monitor and determine the effect that software issues may have on overall F-35 R+M. DOD noted that this process includes a Failure Reporting Analysis Corrective Action System database, a Joint Reliability and Maintainability Evaluation Team, and a Test Data Scoring Board, and uses the Mean-Flight-Hours-Between- Failure metric. Although we agree that the components DOD lists are in place, and that DOD does have a process for assessing F-35 R+M, these actions do not fully address the intent of our recommendation because, as we state in our report, this process is aimed primarily at assessing the reliability of the F-35 aircraft’s hardware and not, specifically, its associated software. Specifically, the process does not determine and monitor software reliability using software-specific metrics and targets. We continue to believe that DOD’s current process and associated metrics do not allow program officials to have a full understanding of the root causes of reliability issues and the extent to which software failures contribute to these issues. As our report states, the reliability of the aircraft is measured with a number of different metrics; however, according to DOD officials, the current process and associated metrics are almost exclusively dedicated to hardware. Since the F-35 is the most software-intensive aircraft DOD has procured to date, establishing a software reliability assessment process with metrics and targets, outside of hardware reliability, would allow DOD to determine the extent to which software failures contribute to the overall R+M of the F-35. If DOD develops such a process, this action should address our recommendation.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to, in order to promote competition, address affordability and, to inform its overarching sustainment strategy, develop a long-term IP Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs. DOD stated that the F-35 acquisition strategy includes a data-management strategy whereby it will continue to assess the acquisition of technical data as specific needs arise throughout the life cycle and will determine the level of specified rights, critical technical data needs and associated costs, through business-case analysis, contract strategy development, and ongoing sustainment planning. If DOD develops a long-term IP strategy that includes the identification of its current level of specified rights, critical technical data needs, and associated costs, this action should address the recommendation.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to clearly document assumptions related to intermediate-level maintenance costs and revise assumptions related to fuel burn rates, part replacement, and depot- maintenance induction in its future F-35 O&S cost estimates. DOD stated that the Program Executive Officer will document intermediate-level maintenance cost assumptions and revise fuel burn rates, part replacement, and depot-maintenance induction assumptions in future annual updates to the F-35 O&S cost estimate. We agree that, if the assumptions are revised and fully documented to reflect the current and future state of the program, this action should address the recommendation.

DOD concurred with the recommendation that the Program Executive Officer conduct uncertainty analysis on future F-35 O&S cost estimates. DOD stated that it will ensure that the Program Executive Officer conducts risk and sensitivity analysis, in accordance with DOD guidance, in future annual updates to the cost estimate. If the Program Executive Officer performs an uncertainty or quantitative risk analysis on future JPO F-35 O&S estimates, this action should address the recommendation.

DOD concurred with the recommendation that CAPE clearly document assumptions related to intermediate-level maintenance and revise its O&S cost estimate’s fuel burn assumptions to better reflect the current and future state of the F-35 program. DOD stated that CAPE’s current documentation of the O&S cost estimate reflects an approved two-level maintenance strategy (i.e., organizational- and depot-level maintenance only, no intermediate-level maintenance). DOD also stated that CAPE will ensure that if the strategy is revised to include any planned intermediate- level maintenance activities, the activities and associated costs will be fully captured within program estimates and documentation. However, the program’s maintenance planning documents already state that intermediate-level maintenance will be used in the sustainment of the F- 35, and it is currently being employed. In its written response, DOD further stated that any intermediate-level costs are already included in the estimate’s mission personnel costs. While DOD guidance states that intermediate-level costs can be combined with unit-level costs, including mission personnel, the guidance also states that this should be noted in the cost estimate’s supporting documentation to avoid an interpretation that a portion of the maintenance costs were omitted. However, as we note in the report, CAPE did not document intermediate-level maintenance costs in the mission personnel or unit-level maintenance sections of the cost estimates. Without clear documentation of these intermediate-level maintenance costs, it is unclear whether they are fully represented in the CAPE estimate. If CAPE clearly documents where and how intermediate maintenance costs are included in its future cost estimates, this action should address the recommendation.

Related to revising its O&S cost estimate’s fuel burn assumptions, DOD stated that CAPE will update its estimate to reflect the latest approved technical baseline and incorporate the latest approved fuel consumption actuals, as this information becomes available. As our report states, the estimate’s fuel burn rate assumption should reflect the most likely future state of the program across its life cycle. To date, fuel consumption actuals are from aircraft flown using limited capabilities—at slower speeds and at lower altitudes than the F-35 will eventually fly—and do not reflect the most likely future state of the program. We agree that, once fuel consumption data from aircraft flown using the full capability are available, these data could be used to update the assumptions. Until then, the estimate’s fuel burn rate assumptions should be revised to reflect the future state of the program. If fully implemented, DOD’s proposed actions to update its estimate should address the recommendation.

Finally, DOD partially concurred with the recommendation that CAPE conduct uncertainty analyses to understand the total potential range of costs and to determine if its estimate reflects the most likely costs associated with the program. DOD stated that the department agrees with the value of understanding potential cost ranges, but stated that it considers the risk and sensitivity analyses regularly performed by CAPE to be a form of uncertainty analysis. As our report states, we acknowledge that CAPE has conducted some sensitivity analyses on previous F-35 O&S estimates, but it has not conducted an uncertainty or quantitative risk analysis on its F-35 O&S cost estimate. According to DOD guidance, sensitivity analyses are useful for identifying critical estimating assumptions, but they have limited utility in providing a comprehensive sense of overall uncertainty. In contrast, an uncertainty or quantitative risk analysis, such as a Monte Carlo simulation, can provide a broad overall assessment of variability in the cost estimate. Similarly, the GAO Cost Estimating and Assessment Guide states that risk and uncertainty analyses should be performed to determine the level of risk associated with an estimate. We continue to believe that without an uncertainty analysis, DOD cannot fully understand how much of the total potential O&S costs are accounted for in its nearly $1 trillion estimate. If CAPE performs an uncertainty analysis on future estimates, this action should address our recommendation.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff has any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To address both of our objectives—that is, to evaluate F-35 sustainment planning and risk and operating and support (O&S) cost estimates—we collected and analyzed information and interviewed officials from the following Department of Defense (DOD) offices:

Office of the Under Secretary of Defense (Acquisitions, Technology

Office of the Assistant Secretary of Defense (Logistics and Materiel

Office of the Director for Cost Assessment and Program Evaluation

Office of the Director for Operational Test and Evaluation (DOT&E);

Office of the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation;

Office of the Deputy Assistant Secretary of Defense for Systems

Department of the Air Force;

Department of the Navy;

Headquarters Marine Corps; and

F-35 Joint Program Office (JPO).

Eglin Air Force Base, Florida;

Marine Corps Air Station Yuma, Arizona;

Naval Air Station Patuxent River, Maryland;

Naval Air Systems Command, Maryland; and

Lockheed Martin, Fort Worth, Texas.

To assess the extent to which DOD has developed a sustainment strategy for the F-35 program and addressed potential risks related to affordability and operational readiness, we reviewed documentation of program plans and analyses with relevant sustainment elements, including the F-35 Life Cycle Sustainment Plan, the Weapon System Planning Document, the F-35 Future Support Construct, the F-35 Autonomic Logistics Global Sustainment Concept of Operations, and the F-35 Operational Requirements Document. We reviewed documentation of ongoing analyses, including Level of Repair Analyses and Business Case Analyses. We compared these documents to DOD acquisition guidance and policy on sustainment planning, including the Defense Acquisition Guidebook, Interim DOD Instruction 5000.02 “Operation of the Defense Acquisition System,” and DOD’s Life Cycle Sustainment Plan and guidance.

To determine the extent to which DOD has developed plans to mitigate sustainment risks, we conducted site visits (see locations above) and interviewed key DOD officials in order to collect information about ongoing operations, testing, and risks to the sustainment of the F-35. After identifying the three most significant risks to the sustainment of the program, we compared DOD’s risk-mitigation efforts to DOD Risk Management Guide and other DOD acquisition guidance and DOD policy and guidance. Specifically, we reviewed DOD’s efforts to monitor and improve the risks of (1) the performance of the Autonomic Logistics Information System (ALIS), (2) the reliability and maintainability (R+M) of the aircraft, and (3) the management of technical data. We then compared these efforts to DOD’s Risk Management Guide, Interim DOD Instruction 5000.02, and other specific guidance and best practices, including DOD’s System of Systems Guide, DOD’s Manual for Improving Reliability, Availability, and Maintainability, and Naval Air Systems Command Software Logistics Primer.

To determine the extent to which DOD has developed a reliable operating and support (O&S) cost estimate for the F-35 program, we evaluated both DOD’s Joint Program Office (JPO) and its Cost Assessment and Program Evaluation (CAPE) office 2013 O&S cost estimates using GAO’s Cost Estimating and Assessment Guide. Specifically, the methodology outlined in the Cost Estimating and Assessment Guide is a compilation of best practices that federal cost-estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life cycle of an acquisition program. The JPO’s estimate is used to drive budget planning and CAPE’s estimate is used to assess affordability and is required at major milestones or by special request. Generally, either the JPO’s or CAPE’s estimate is used as DOD’s estimate of record and published in the Selected Acquisition Report (SAR)—a report that DOD is generally required to submit annually to Congress on each of its major defense acquisition programs. For the F-35, DOD decided to use the CAPE O&S estimate in the 2012 and 2013 SARs. We evaluated both the JPO and the CAPE O&S estimates because both estimates are used in DOD decision making. According to the GAO Cost Estimating and Assessment Guide, there are four general characteristics of sound cost estimating: being well-documented, comprehensive, accurate, and credible. For the purposes of this engagement, we conducted a limited assessment and evaluated two of these characteristics: comprehensive and credible. We chose to evaluate the estimates’ comprehensiveness because if a cost estimate is not comprehensive then it cannot fully meet the characteristics of being well-documented or accurate. For example, if the cost estimate is missing some cost elements, then the documentation will be incomplete and the estimate will be inaccurate. We also included an assessment of the credible characteristic due to the substantial estimated cost of the program and the need to better understand and quantify the F-35 O&S cost risk. To determine whether the comprehensive and credibility characteristics were met, we reviewed CAPE and JPO cost-estimating documentation, including data sources, assumptions, and calculations, and we interviewed cost-estimating officials from these offices. In addition, to determine how much more frequently parts were being replaced than assumed in the JPO cost estimate (see table 2), we compared the mean flight hour between removals (MFHBR) data for the Air Force’s F-35A (CTOL) and the Marine Corps’ F-35B (STOVL) from a 12-month period ending in March 31, 2014, to the MFHBR assumptions used in the JPO estimate.

We also gathered F-35 reliability and maintainability (R+M) data2013 through 2014 that had been verified through DOD’s Joint Reliability and Maintainability Evaluation Team (JRMET). To determine the reliability of these data, we collected information on how the data were collected, managed, and used through a survey and interviews with relevant DOD officials. In addition to the data-reliability survey, we also reviewed user manual and related documentation to determine the limitations of the data. By assessing this information against GAO data-quality standards, we determined that the data presented in our findings were sufficiently from reliable for presenting information about the aircraft’s reliability in this report.

We conducted this performance audit from August 2013 through September 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Scoring of 2013 F-35 Operating and Support (O&S) Estimates

We completed an overall assessment of the Joint Program Office (JPO) and Cost Assessment and Program Evaluation (CAPE) F-35 O&S estimates on the basis of two characteristics—comprehensiveness and credibility—and their associated best practices derived from the GAO Cost Estimating and Assessment Guide. After reviewing documentation that JPO and CAPE submitted for their 2013 F-35 O&S cost estimates, conducting interviews with JPO and CAPE cost-estimating officials, and reviewing relevant sources, we determined that these cost estimates are not fully reliable. While the estimates are comprehensive, not all of the assumptions that the estimates were based on were clearly documented or reasonable. In addition, the estimates are not fully reliable in that the JPO estimate is partially credible and the CAPE estimate is minimally credible. These evaluations are shown in table 3 below. We determined the overall assessment rating by assigning each individual best practice rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the average of the individual best practice assessment ratings to determine the overall rating for each of the two characteristics. The resulting average becomes the Overall Assessment as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered reliable if the overall assessment ratings for each of the two characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: Alissa Czyz, Assistant Director; Jennifer Echard; Jeffrey Hubbard; Abishek Krupanand; Jason Lee; Jennie Leotta; Katie Mauldin; Carol Petersen; Amie Steele; Alyssa Weir; Cheryl Weissman; and Amanda Weldon.

Related GAO Products

Weapon Systems Management: DOD Has Taken Steps to Implement Product Support Managers but Needs to Evaluate Their Effects. GAO-14-326. Washington, D.C.: April 29, 2014.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-14-340SP. Washington, D.C.: March 31, 2014.

F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014.

F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014.

F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013.

F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-13-294SP. Washington, D.C.: March 28, 2013.

F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013.

Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012.

Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012.

Tactical Aircraft: F-22A Modernization Program Faces Cost, Technical, and Sustainment Risks. GAO-12-447.Washington, D.C.: May 2, 2012.

Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012.

Defense Logistics: Improvements Needed to Enhance Oversight of Estimated Long-term Costs for Operating and Supporting Major Weapon Systems. GAO-12-340. Washington, D.C.: February 2, 2012.

Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011.

Defense Acquisition: DOD Should Clarify Requirements for Assessing and Documenting Technical-Data Needs. GAO-11-469. Washington, D.C.: May 11, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.

Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.

Defense Management: DOD Needs to Monitor and Assess Corrective Actions Resulting from Its Corrosion Study of the F-35 Joint Strike Fighter. GAO-11-171R. Washington, D.C.: December 16, 2010.

Tactical Aircraft: DOD’s Ability to Meet Future Requirements Is Uncertain, with Key Analyses Needed to Inform Upcoming Investment Decisions. GAO-10-789. Washington, D.C.: July 29, 2010.

Joint Strike Fighter: Significant Challenges and Decisions Ahead. GAO-10-478T.Washington, D.C.: March 24, 2010.

Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting Warfighter Requirements on Time. GAO-10-382. Washington, D.C.: March 19, 2010.

Joint Strike Fighter: Significant Challenges Remain as DOD Restructures Program. GAO-10-520T. Washington, D.C.: March 11, 2010.

Joint Strike Fighter: Strong Risk Management Essential as Program Enters Most Challenging Phase. GAO-09-711T. Washington, D.C.: May 20, 2009.