F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments
Released: 2017-04-24T00:00:00

Letter

Background

As we have previously reported, DOD began the F-35 acquisition program in October 2001 without adequate knowledge about the aircraft’s critical technologies or design. In addition, DOD’s acquisition strategy called for high levels of concurrency or overlap among development, testing, and production. In our prior work, we have identified the lack of adequate knowledge and high levels of concurrency as major drivers of the significant cost and schedule growth as well as performance shortfalls that the program has experienced since 2001. The program has been restructured three times since it began: first in December 2003, again in March 2007, and most recently in March 2012. The most recent restructuring was initiated in early 2010 when the program’s unit cost estimates exceeded critical thresholds established by statute—a condition known as a Nunn-McCurdy breach. DOD subsequently certified to Congress in June 2010 that the program was essential to national security and needed to continue. DOD then began efforts to significantly restructure the program and establish a new acquisition program baseline. These restructuring efforts continued through 2011 and into 2012, during which time the department increased the program’s cost estimates and extended its testing and delivery schedules. Since then costs have remained relatively stable. Table 1 shows the cost, quantity, and schedule changes from the initial program baseline and the relative stability since the new baseline was established.

As the program has been restructured, DOD has also reduced near-term aircraft procurement quantities. From 2001 and through 2007, DOD deferred the procurement of 931 aircraft into the future, and then again from 2007 and through 2012, DOD deferred another 450 aircraft. Figure 1 shows how planned quantities in the near term steadily declined over time.

The F-35 is DOD’s most costly acquisition program, and over the last several years we have reported on the affordability challenges facing the program. As we reported in April 2016, the estimated total acquisition cost for the F-35 program was $379 billion, and the program would require an average of $12 billion per year from 2016 through 2038. The program expects to reach peak production rates for U.S. aircraft in 2022, at which point DOD expects to spend more than $14 billion a year on average for a decade (see fig. 2). Given these significant acquisition costs, we found that DOD would likely face affordability challenges as the F-35 program competes with other large acquisition programs, including the B-21 bomber, KC-46A tanker, and Ohio Class submarine replacement. In addition, in September 2014, we reported that DOD’s F-35 sustainment strategy may not be affordable.

Through 2016, DOD had awarded contracts for production of 9 lots of F- 35 aircraft, totaling 285 aircraft (217 aircraft for the U.S. and 68 aircraft for international partners or foreign military sales). At the time of this report, the contract for lot 10 had not been signed.

In 2013, the Departments of the Navy and the Air Force issued a joint report to the congressional defense committees providing that the Marine Corps and Air Force would field initial operating capabilities in 2015 and 2016, respectively, with aircraft that had limited warfighting capabilities. The Navy did not plan to field its initial operating capability until 2018, after the F-35’s full warfighting capabilities had been developed and tested. These dates represented a delay of 5 to 6 years from the program’s initial baseline. As planned, the Marine Corps and Air Force declared initial operational capability (IOC) in July 2015 and August 2016, respectively.

Continuing Problems with Flight Testing of Software Capabilities Will Delay the Completion of Development and Increase Development Costs

DOD will need more time and money than expected to complete the remaining 10 percent of the F-35 development program. DOD has experienced delays in testing the software and systems that provide warfighting capabilities, known as mission systems, largely because the software has been delivered late to be tested and once delivered has not worked as expected. Program officials have had to regularly divert resources from developing and testing of more advanced software capabilities to address unanticipated problems with prior software versions. These problems have compounded over time, and this past year was no exception. DOD began testing the final block of software— known as block 3F—later than expected, experienced unanticipated problems with the software’s performance, and thus did not complete all mission systems testing it had planned for 2016. As a result, the F-35 program office has noted that more time and money will be needed to complete development. The amount of time and money could vary significantly depending on the program’s ability to complete developmental and operational testing. We estimate that developmental testing could be delayed as much as 12 months, thus delaying the start of initial operational testing, and total development costs could increase by nearly $1.7 billion. In addition, the Navy’s IOC and the program’s full-rate production decision could also be delayed.

Problems with Mission Systems Software Continue to Cause Delays in Flight Testing

DOD continues to experience delays in F-35 mission systems testing. Although mission systems testing is about 80 percent complete, the complexity of developing and testing mission systems has been troublesome. For the F-35 program, DOD is developing and fielding mission systems capabilities in software blocks: (1) Block 1, (2) Block 2A, (3) Block 2B, (4) Block 3i, and (5) Block 3F. Each subsequent block builds on the capabilities of the preceding block. Over the last few years, program officials have had to divert resources—personnel and infrastructure—from developing and testing of more advanced software blocks to address unanticipated problems with prior software blocks. Over time, this practice has resulted in compounding delays in mission systems testing. Blocks 1 through 3i are now complete, and the program is currently focused on developing and testing Block 3F, the final software block in the current development program. Figure 3 illustrates the mission systems software blocks being developed for the program, the percentage of test points completed by block, and the build-up to full warfighting capability with Block 3F.

Program officials spent some of 2016 addressing problems with Block 3i mission systems unexpectedly shutting down and restarting—an issue known as instability—which delayed Block 3F testing. In early 2016, officials were developing and testing Block 3i concurrently with Block 3F. In order to ensure that the Block 3i instability was addressed in time for the Air Force’s planned IOC in August 2016, officials diverted resources from Block 3F.That decision delayed subsequent testing that had been planned for Block 3F. Further delays resulted from the discovery of instability and functionality problems with Block 3F. To mitigate some schedule delays, program officials implemented a new process to introduce software updates quicker than normal. Although the quick software releases helped to ensure that testing continued, the final planned version of Block 3F, which was originally planned to be released to testing in February 2016, was not released until late November 2016, nearly a 10-month delay. As a result, program officials have identified the need for additional time to complete development. Program officials now project that developmental testing, which was expected to be completed in May, will conclude in October 2017, 5 months later than planned.

However, based on our analysis, the program’s projection is optimistic as it does not reflect historical F-35 test data. Program officials believe that going forward they may be able to devote more resources to mission systems testing, which could lead to higher test point completion rates than they have achieved in the past. According to GAO best practices, credible schedule estimates are rooted in historical data. As of November 2016, program officials estimated that the program will need to complete as much as an average of 384 mission systems test points per month in order to finish flight testing by October 2017—a rate that the program has rarely achieved before. Our analysis of historical test point data as of December 2016 indicates that the average test point execution rates are much lower, at 220 mission systems test points per month. In addition, historical averages suggest that test point growth—additions to the overall test points from discovery in flight testing—is much higher than program officials assume, while estimated deletions—test points that are considered no longer required—are lower than assumed. Using the historical F-35 averages, we project that developmental testing may not be completed until May 2018, a 12-month delay from the program’s current plan. Table 2 provides a comparison of the assumptions used to determine delays in developmental testing.

Our estimation of delays in completing developmental testing does not include the time it may take to address the significant number of existing deficiencies. The Marine Corps and Air Force declared IOC with limited capability and with several deficiencies. As of October 2016, the program had more than 1,200 open deficiencies, and senior program and test officials deemed 276 of those critical or of significant concern to the military services. Several of the critical deficiencies are related to the aircraft’s communications, data sharing, and target tracking capabilities. Although the final planned version of Block 3F software was released to flight testing in November 2016 and contained all 332 planned warfighting capabilities, not all of those capabilities worked as intended. In accordance with program plans, it was the first time some of the Block 3F capabilities had been tested. According to a recent report by the Director, Operational Test and Evaluation (DOT&E), fixes for less than half of the 276 deficiencies were included in the final planned version of Block 3F software. Prime contractor officials stated that additional software releases will likely be required to address deficiencies identified during the testing of the final planned version of Block 3F software, but they do not yet know how many releases will ultimately be needed.

Delays in Developmental Testing Will Likely Affect Plans for Initial Operational Testing, Navy Operational Capability Dates, and Full-Rate Production

Delays in developmental testing will likely drive delays in current plans to start F-35 initial operational test and evaluation. Program officials have noted that according to their calculations developmental testing will end in October 2017 and initial operational testing will begin in February 2018. However, DOT&E officials, who approve operational test plans, anticipate that the program will more likely start operational testing in late 2018 or early 2019, at the earliest. Figure 4 provides an illustration of the current program schedule and DOT&E’s projected delays.

DOT&E’s estimate for the start of initial operational testing is based on the office’s projection that developmental testing will end in July 2018 and that retrofits needed to prepare the aircraft for operational testing will not be completed until late 2018 at the earliest. There are 23 aircraft—many of which are early production aircraft—that require a total of 155 retrofits before they will be ready to begin operational testing. As of January 2017, 20 of those retrofits were not yet under contract, and program officials anticipated some retrofits would be completed in late 2018.

To mitigate possible schedule delays, program officials are considering a phased start to operational testing. However, current program test plans require training and preparation activities before initial operational test and evaluation begins. Those activities, as outlined in the test plan, are expected to take approximately 6 months. Changes to this approach would require approval from DOT&E. According to DOT&E officials, however, the program has not yet provided any detailed strategy for implementing a new approach or identified a time frame for revising the test plan.

Significant delays in initial operational testing will likely affect two other upcoming program decisions: (1) the Navy’s decision to declare IOC and (2) DOD’s decision to begin full-rate production. In a 2015 report to the congressional defense committees, the Under Secretary of Defense for Acquisition, Technology and Logistics stated that the Navy’s IOC declaration is on track for February 2019 pending completion of initial operational test and evaluation. If initial operational testing does not begin until February 2019 as the DOT&E predicts, the Navy may need to consider postponing its IOC date. Likewise, DOD’s full-rate production decision, currently planned for April 2019, may have to be delayed. According to statute, a program may not proceed beyond low-rate initial production into full-rate production until initial operational test and evaluation is completed and DOT&E has submitted to the Secretary of Defense and the congressional defense committees a report that analyzes the results of operational testing. If testing does not begin until February 2019 and takes 1 year, as expected, DOD will not have the report in time to support a full-rate production decision by April 2019.

Delays in Development Will Increase Program Costs and Add to Concurrency

The current delays in F-35 developmental testing will also result in increased development costs. Based on the program office’s estimate of a 5-month delay in developmental testing, the F-35 program will need an additional $532 million to complete the development contract. According to GAO best practices, credible cost estimates are also rooted in historical data. Using historical contractor cost data from April 2016 to September 2016, we calculated the average monthly cost associated with the development contract. If developmental testing is delayed 12 months, as we estimate, and operational testing is not completed until 2020, as projected by DOT&E, then we estimate that the program could need more than an additional $1.7 billion to complete the F-35 development contract. Similarly, the Cost Assessment and Program Evaluation office within the Office of the Secretary of Defense has estimated that the program will likely need more than $1.1 billion to complete the development contract. In these estimates, the majority of the additional funding would be needed in fiscal year 2018. Specifically, program officials believe that an additional $353.8 million may be needed in fiscal year 2018, while we estimate that they could need more than three times that amount— approximately $1.3 billion—as illustrated in figure 5.

The program plans to fund their estimated development program deficit through several means. For example, although the program office 2018 preliminary budget projection reflected a reduction of $81 million in development funding over the next few years, as compared to DOD’s fiscal year 2017 budget request, program officials expect DOD to restore this reduction in its official fiscal year 2018 budget request. In addition, program officials plans to increase the budget request, as compared to their fiscal year 2017 budget request, for development funding in fiscal years 2018, 2019, and 2020 by $451 million and likewise reduce their budget request for procurement funding over those years. To make up for the reduction in requested procurement funding, the program plans to reprogram available procurement funds appropriated in prior fiscal years. Any additional funding beyond $451 million would likely have to come from some other source. Figure 5 compares DOD’s and our estimates for development funding needs from fiscal years 2018 through 2021.

As developmental testing is delayed and DOD procures more aircraft every year, concurrency costs—the costs of retrofitting delivered aircraft—increase. For example, from 2015 to 2016, the program experienced a $70 million increase in concurrency costs. This increase was partially driven by the identification of new technical issues found during flight testing that were not previously forecasted, including problems with the F-35C outer-wing structure and F-35B landing gear. Problems such as these have to be fixed on aircraft that have already been procured. Thus far, DOD has procured 285 aircraft and has experienced a total of $1.77 billion in concurrency costs. Although testing is mostly complete, any additional delays will likely result in delays in the incorporation of known fixes, which would increase the number of aircraft that will require retrofits and rework and further increase concurrency costs as more aircraft are procured. According to program officials, most of the retrofits going forward are likely to be software related and thus less costly. However, according to DOD’s current plan, 498 aircraft will be procured by the time initial operational testing is complete. If the completion of operational testing is delayed to 2020, as DOT&E predicts, the number of procured aircraft will increase to 584 as currently planned, making 86 additional aircraft subject to any required retrofits or rework.

DOD’s Planned Investments in Modernization and Economic Order Purchase May Be Premature

In fiscal year 2018, F-35 program officials expect to invest more than $1.2 billion to start two efforts while simultaneously facing significant shortfalls in completing the F-35 baseline development program, as discussed above. Specifically, DOD and program officials project that in fiscal year 2018 the program will need over $600 million to begin development of follow-on modernization of the F-35 and more than $650 million to procure economic order quantities (EOQ) of parts to achieve cost savings during procurement. Contracting for EOQ generally refers to the purchase of parts in larger more economically efficient quantities to minimize the cost of these items. DOD officials emphasized that the specific amount of funding needed for these investments could change as the department finalizes its fiscal year 2018 budget request. Regardless, these investments may be premature. Early Block 4 requirements, which represent new capabilities beyond the original requirements, may not be fully informed before DOD plans to solicit proposals from contractors for how they might meet the government’s requirements—a process known as request for proposal (RFP). According to DOD policy, the Development RFP Release Decision Point is the point at which a solid business case is formed for a new development program. Until Block 3F testing is complete, DOD will not have the knowledge it needs to develop and present an executable business case for Block 4, with reliable cost and funding estimates.

Follow-on Modernization May Be Too Early

Due to evolving threats and changing warfighting environments, program officials project that the program will need over $600 million in fiscal year 2018 to award a contract to begin developing new F-35 capabilities, an effort referred to as follow-on modernization. However, the requirements for the first increment of that effort, known as Block 4, have not been finalized. Block 4 is expected to be developed and delivered in four phases—currently referred to as 4.1, 4.2, 4.3, and 4.4. Program officials expect phases 4.1 and 4.3 to be primarily software updates, while 4.2 and 4.4 consist of more significant hardware changes. The program has drafted a set of preliminary requirements for Block 4 that focused on the top-level capabilities needed in phases 4.1 and 4.2, but the requirements for the final two phases have not been fully defined. In addition, as of January 2017, these requirements had not been approved by the Joint Requirements Oversight Council.

Delays in developmental testing of Block 3F are also likely to affect Block 4 requirements. DOD policy states that requirements are to be approved before a program reaches the Development RFP Decision Point in the acquisition process. GAO best practices emphasize the importance of matching requirements and resources in a business case before a development program begins. For DOD, the Development RFP Release Decision Point is the point at which plans for the program must be most carefully reviewed to ensure that all requirements have been approved, risks are understood and under control, the program plan is sound, and the program will be affordable and executable. Currently, F-35 program officials plan to release the RFP for Block 4.1 development in the third quarter of fiscal year 2017, nearly 1 year before we estimate Block 3F developmental testing will be completed. Program officials have stated that Block 3F is the foundation for Block 4, but continuing delays in Block 3F testing make it difficult to fully understand Block 3F functionality and its effect on early Block 4 capabilities. If new deficiencies are identified during the remainder of Block 3F testing, the need for new technologies may arise, and DOD may need to review Block 4 requirements again before approving them.

Costs and Benefits of Economic Order Quantity Purchases Are Not Clear

In April 2016, we reported that the F-35 program office was considering what it referred to as a block buy contracting approach that we noted had some potential economic benefits but could limit congressional funding flexibility. The program office has since changed its strategy to consist of contracts for EOQ of 2 years’ worth of aircraft parts followed by a separate annual contract for procurement of lot 12 aircraft with annual options for lots 13 and 14 aircraft. Each of these options would be negotiated separately, similar to how DOD currently negotiates contracts.

As of January 2017, details of the program office’s EOQ approach were still in flux. In 2015, the program office contracted with RAND Corporation to conduct a study of the potential cost savings associated with several EOQ approaches. According to the results of that study, in order for the government to get the greatest benefit, the aircraft and engine contractors would need to take on risk by investing in EOQ on behalf of the department in fiscal year 2017. Program officials envision that under this arrangement the contractors would be repaid by DOD at a later date. However, as of January 2017, contractors stated they were still negotiating the terms of this arrangement; therefore, the specific costs and benefits remained uncertain. Despite this uncertainty, the program office plans to seek congressional approval to make EOQ purchases and expects to need more than $650 million for that purpose in fiscal year 2018. Program officials believe that this upfront investment would result in a significant savings over the next few years for the U.S. services. However, given the uncertainties around the level of contractor investment, it is not clear whether an investment of more than $650 million, if that is the final amount DOD requests in fiscal year 2018, will be enough to yield significant savings. Regardless, with cost growth and schedule delays facing the F-35 baseline development program, it is unclear whether DOD can afford to fund this effort at this time. According to internal control standards, agencies should communicate with external stakeholders, such as Congress. With a potential investment of this size, particularly in an uncertain budget environment, it is important that program officials finalize the details of this approach before asking for congressional approval and provide Congress with a clear understanding of the associated costs to ensure that funding decisions are fully informed.

Aircraft Manufacturing and Reliability Are Improving in Some Areas

The F-35 airframe and engine contractors continue to report improved manufacturing efficiency, and program data indicate that reliability and maintainability are improving in some areas. Over the last 5 years, the number of U.S. aircraft produced and delivered by Lockheed Martin has increased, and manufacturing efficiency and quality have improved over time. Similarly, manufacturing efficiency and quality metrics are improving for Pratt & Whitney. Although some engine aircraft reliability and maintainability metrics are not meeting program expectations, there has been progress in some areas, and there is still time for further improvements.

Airframe and Engine Manufacturing Efficiency Is Improving, but Supply Chain Challenges Remain

Overall the airframe manufacturer, Lockheed Martin, is improving efficiency and product quality. Over the last 5 years, the number of aircraft produced and delivered by Lockheed Martin has increased from 29 aircraft in 2012 to 46 aircraft in 2016. Since 2011, a total of 200 production aircraft have been delivered to DOD and international partners, 46 of which were delivered in 2016. As of January 2017, 142 aircraft were in production, worldwide. As more aircraft are delivered, the number of labor hours needed to manufacture each aircraft declines. Labor hours decreased from 2015 to 2016, indicating production maturity. In addition, instances of production line work done out of sequence remains relatively low, with the exception of an increase at the end of 2016 due to technical issues, such as repairing coolant tube insulation (see app. III). Further, the number of quality defects and total hours spent on scrap, rework, and repair declined in 2016.

Although data indicate that airframe manufacturing efficiency and quality continue to improve, supply chain challenges remain. Some suppliers are delivering late and non-conforming parts, resulting in production line inefficiencies and workarounds. For example, in 2016, Lockheed Martin originally planned to deliver 53 aircraft, but quality issues with insulation on the coolant tubes in the fuel tanks resulted in the contractor delivering 46 aircraft. According to Lockheed Martin officials, late deliveries of parts are largely due to late contract awards and supply base capacity. While supplier performance is generally improving, it is important for suppliers to be prepared for both production and sustainment support going forward. Inefficiencies, such as conducting production line work out of sequence, could be exacerbated if late delivery of parts continues as production more than doubles over the next 5 years.

The engine manufacturer, Pratt & Whitney, is also improving efficiency. As of October 2016, Pratt & Whitney had delivered 279 engines. The labor hours required to assemble an F-35 engine decreased quickly and has remained relatively steady since around the 70th engine produced, and little additional efficiency is expected to be gained. Other Pratt & Whitney manufacturing metrics indicate that production efficiency and quality are improving. Scrap, rework, and repair costs were reduced from 2.22 percent in 2015 to 1.8 percent in 2016. We previously reported that according to Pratt & Whitney officials, moving from a hollow blade design to a solid blade would reduce scrap and rework costs because it is easier to produce. However, Pratt & Whitney experienced unanticipated problems with cracking in the solid blade design. As a result, Pratt & Whitney is continuing to produce a hollow blade while it further investigates the difficulty and costs associated with a solid blade design. Pratt & Whitney’s supply chain continues to make some improvements. For example, critical parts are being delivered ahead of schedule, and some are already achieving 2017 rate requirements. To further ensure that suppliers are capable of handling full-rate production, Pratt & Whitney is pursuing the potential to have multiple suppliers for some engine parts, which officials believe will help increase manufacturing capacity within the supply chain.

Aircraft Reliability and Maintainability Metrics Have Improved in Some Areas

Although the program has made progress in improving system-level reliability and maintainability, some metrics continue to fall short of program expectations in several key areas. For example, as shown in figure 6, while metrics in most areas were overall trending in the right direction, the F-35 program office’s internal assessment indicated that as of August 2016 the F-35 fleet was falling short of reliability and maintainability expectations in 11 of 21 areas.

Although many of the metrics remain below program expectations, some of the metrics have shown improvement over the last year, and time remains for continued improvements. For example, our analysis indicates that since 2015, the F-35A reliability has improved from 4.3 mean flight hours between failure attributable to design issues to 5.7 hours, nearly achieving the goal at system maturity of 6 hours. The F-35A mean flight hours between maintenance event metric has also improved and is now meeting program expectations. As of August 2016, the F-35 fleet had only flown a cumulative total of 63,187 flight hours. The program has time for further improvement as the ultimate goals for these reliability and maintainability metrics are to be achieved by full system maturity, or 200,000 cumulative flight hours across the fleet. The program also plans to improve these metrics through additional design changes.

Engine reliability varied in 2016. In April 2016, we reported that Pratt & Whitney had implemented a number of design changes that resulted in significant improvements to one reliability metric: mean flight hours between failure attributable to design issues. At the time of our report, contractor data indicated the F-35A and F-35B engines were at about 55 percent and 63 percent, respectively, of where the program expected them to be. According to contractor data as of September 2016, the program was unable to achieve a significant increase in reliability over the last year, which left the F-35A and F-35B engines further below expectations—at about 43 percent and 41 percent, respectively. Other reliability metrics such as engine’s impact on aircraft availability, engine maintenance man-hours, and the time between engine removals are meeting expectations. On average, from June 2016 through November 2016, the engine affected only about 1.47 percent of the overall aircraft availability rates, and none of the top 30 drivers affecting aircraft availability were related to the engine. According to Pratt & Whitney officials, the F-35 engine requires fewer maintenance man-hours per flight hour than legacy aircraft, and engines for the F-35A and F-35B are currently performing better than required for the average number of flight hours between engine removals. Program and contractor officials continue to identify ways to further improve reliability through a number of design changes and expect reliability to continue to improve lot over lot.

Conclusions

As the F-35 program approaches the end of development, its schedule and cost estimates are optimistic. The program’s cost and schedule estimates to complete development are hundreds of millions of dollars below and several months under other independent estimates, including our own. If the program experiences schedule delays as we predict, it could require a total of nearly $1.5 billion in fiscal year 2018 alone. However, program officials project that the program will only need $576.2 million in fiscal year 2018 to complete baseline development. At the same time, program officials expect that more than $1.2 billion could be needed to commit to Block 4 and EOQ in fiscal year 2018. DOD must prioritize funding for the baseline development program over the program office’s desire for EOQ and Block 4. If baseline development is not prioritized and adequately funded, and costs increase as predicted by GAO and others, then the program will have less recourse for action and development could be further delayed. In addition, with baseline development still ongoing the program will not likely have the knowledge it needs to present a sound business case for soliciting contractor proposals for Block 4 development in fiscal year 2017. Although Block 4 and EOQ may be desirable, prioritizing funding for these efforts may not be essential at this time. Prioritizing funding for baseline development over these two efforts would ensure that the program has the time and money needed to properly finish development and thus lay a solid knowledge-based foundation for future efforts.

Recommendations for Executive Action

To ensure that DOD adequately prioritizes its resources to finish F-35 baseline development and delivers all of the promised warfighting capabilities and that Congress is fully informed when making fiscal year 2018 budget decisions, we are making the following three recommendations to the F-35 program office through the Secretary of Defense. 1. Reassess the additional cost and time needed to complete developmental testing using historical program data. 2. Delay the issuance of the Block 4 development request for proposals at least until developmental testing is complete and all associated capabilities have been verified to work as intended. 3. Finalize the details of DOD and contractor investments associated with an EOQ purchase in fiscal year 2018, and submit a report to Congress with the fiscal year 2018 budget request that clearly identifies the details, including costs and benefits of the finalized EOQ approach.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. DOD’s comments are reprinted in appendix IV and summarized below. DOD also provided technical comments, which were incorporated as appropriate.

DOD did not concur with our recommendation to reassess the additional cost and time needed to complete developmental testing using historical program data. DOD stated that it will continue to assess the assumptions and decisions made, and communicate any necessary adjustments relative to both cost and time needed to complete developmental testing.

DOD also stated that it had considered historical data in its assessment and concluded that developmental testing could extend into February 2018. While this possible slip is noted in our report, it is unclear to us the extent to which the data underpinning DOD’s assessment reflected the program’s historical averages. While the program’s analysis that we examined did reflect test point accomplishment rates that were more aligned with what the program achieved in 2016 (i.e. around 290 points per month) those rates were still higher than the historical average. Other key inputs to that analysis also differed significantly from the program’s historical averages. For example, program officials assumed only a 42 percent test point growth rate when the program’s historical average test point growth was 63 percent, and in 2016 alone the test point growth rate was 115 percent. Several other DOD officials have identified possible delays beyond February 2018. In a memo sent to Congress in December 2016, the Under Secretary of Defense for Acquisition, Technology and Logistics stated that developmental testing could go as long as May 2018, and DOT&E analysis also indicates that developmental testing may not conclude until mid-2018. We continue to believe that our recommendation is valid.

DOD also did not concur with our recommendation to delay the issuance of the Block 4 development request for proposals until developmental testing is complete. According to DOD, delaying the request for proposals could unnecessarily delay delivery of needed capabilities to the warfighters. However, as program officials stated, Block 3F software establishes the foundation for Block 4. Therefore, continuing delays in Block 3F testing will likely make it difficult to fully understand Block 3F functionality and its effect on early Block 4 requirements. If new deficiencies are identified during the remainder of Block 3F testing, the need for new technologies may arise, and DOD may need to review Block 4 requirements again before approving them which could lead to additional delays. Therefore, we continue to believe that our recommendation is valid.

DOD stated that it partially concurred with our third recommendation to finalize the details of investments associated with an EOQ purchase in fiscal year 2018, and submit a report to Congress with the fiscal year 2018 budget request that clearly identifies those details. However, in its response, the department outlined steps that address it. For example, DOD stated that it had finalized the details of DOD and contractor investments associated with an EOQ purchase and will brief Congress on the details, including costs and benefits of the finalized EOQ approach.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; and the Under Secretary of Defense for Acquisition, Technology and Logistics. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Prior GAO Report and DOD Actions

Appendix II: Scope and Methodology

To assess the F-35 program’s remaining development and testing we interviewed officials from the program office and contractors—Lockheed Martin and Pratt & Whitney. We obtained and analyzed data on mission systems test point execution, both planned and accomplished from 2011 through 2016 to calculate historical test point averages per month. We compared test progress against the total program requirements to determine the number of test points that were completed and remaining as of December 2016. We used the average test point rate based on the historical data to determine the number of months needed to complete the remaining test points. To identify the program’s average monthly costs, we analyzed contractor cost performance data from April 2016 through September 2016 to identify average contract costs per month. Using a 12-month delay and the average contract costs per month, we calculated the costs to complete developmental testing. In order to determine costs to complete development, we first determined the percent change, year to year, in the program office’s development funding requirement estimate from 2018 to 2021. We then reduced our estimate using those percentages from 2018 to 2021. We discussed key aspects of F-35 development progress, including flight testing progress, with program management and contractor officials as well as DOD test officials and program test pilots. To assess the reliability of the test and cost data, we reviewed the supporting documentation and discussed the development of the data with DOD officials instrumental in producing them. In addition, we interviewed officials from the F-35 program office, Lockheed Martin, Pratt & Whitney, and the Director, Operational Test and Evaluation office to discuss development test plans, achievements, and test discoveries.

To assess DOD’s proposed plans for future F-35 investments, we discussed cost and manufacturing efficiency initiatives, such as the economic order quantities approach, with contractor and program office officials to understand potential cost savings and plans. To assess the program’s follow-on modernization plans, we discussed the program’s plans with program office officials. We reviewed the fiscal year 2017 budget request to identify costs associated with the effort. We also reviewed and analyzed best practices identified by GAO and reviewed relevant DOD policies and statutes. We compared the acquisition plans to these policies and practices.

To assess ongoing manufacturing and supply chain performance, we obtained and analyzed data related to aircraft delivery rates and work performance data from January 2016 to December 2016. These data were compared to program objectives identified in these areas and used to identify trends. We reviewed data and briefings provided by the program office, Lockheed Martin, Pratt & Whitney, and the Defense Contract Management Agency in order to identify issues in manufacturing processes. We discussed reasons for delivery delays and plans for improvement with Lockheed Martin and Pratt & Whitney. We collected and analyzed data related to aircraft quality through December 2016. We collected and analyzed supply chain performance data and discussed steps taken to improve quality and deliveries with Lockheed Martin and Pratt & Whitney. We also analyzed reliability and maintainability data and discussed these issues with program and contractor officials.

We assessed the reliability of DOD and contractor data by reviewing existing information about the data and interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report.

We conducted this performance audit from June 2016 to April 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Status of F-35 Technical Risks

As developmental testing nears completion, the F-35 program continues to address technical risks. The program has incorporated design changes that appear to have mitigated several of the technical risks that we have highlighted in prior reports, including problems with the arresting hook system and bulkhead cracks on the F-35B. However, over the past year, the program continued to address risks with the Helmet Mounted Display, Autonomic Logistics Information System (ALIS), the ejection seat and engine seal that we have identified in the past. The program also identified new risks with the F-35C wing structure and catapult launches, and coolant tube insulation. The status of the Department of Defense’s (DOD) efforts to address these issues is as follows: Helmet Mounted Display: A new helmet intended to address shortfalls in night vision capability, among other things, was developed and delivered to the program in 2015. Developmental testing of the new helmet is mostly complete, and officials believe that issues such as latency and jitter have been addressed. Green glow, although improved, continues to add workload for the pilots when landing at sea. Officials believe that they have done as much as they can to fix the green glow problems with the hardware currently available.

ALIS: ALIS continues to lack required capabilities; for instance, engine parts information is not included in the current version of ALIS, although it is expected to be completed in the spring of 2017. In 2016, officials began testing ALIS in an operational environment which has led to some improvements. However, capabilities, including the prognostics health management downlink, have been deferred to follow-on modernization. In 2016, officials acknowledged compounding development delays and restructured the development schedule for ALIS. The new schedule shows that some capabilities that were planned in the earlier versions of ALIS will now be deferred to later versions. In April 2016, we reported that F-35 pilots and maintainers identified potential functionality risks to ALIS and that DOD lacked a plan to address these risks as key milestone dates approached, which could result in operational and schedule risks.

Engine seal: Officials have identified a design change to address the technical problem that resulted in an engine fire in June 2014. This design change was validated and incorporated into production in 2015. Engine contractor officials identified 194 engines that needed to be retrofitted, and as of October 2016, 189 of those retrofits had been completed. The engine contractor, Pratt & Whitney, is paying for these retrofits.

Ejection seat: In 2015, officials discovered that pilots who weigh less than 136 pounds could possibly suffer neck injuries during ejection. Officials stated that the risk of injury is due to the over-rotation of the ejection seat in combination with the thrust from the parachute deployment during ejection. Officials noted that although the problem was discovered during testing of the new Helmet Mounted Display, the helmet’s weight was not the root cause. The program has explored a number of solutions to ensure pilot safety including installing a switch for light-weight pilots that would slow the release of the parachute deployment, installing a head support panel that would reduce head movement, and reducing the weight of the helmet. The final design completed qualification testing in 2016 and is expected to be incorporated into production lot 10. The cost of these changes has not yet been determined.

F-35C outer-wings: In 2016, officials identified structural issues on the F- 35C outer-wing when carrying an AIM-9X missile. In order to resume the test program, officials identified a design change to include strengthening the wings’ material that was incorporated onto a test aircraft. Officials expect to incorporate retrofits to delivered aircraft by 2019 and will incorporate changes into production in lot 10.

F-35C catapult launches: In 2016, officials identified issues with violent, uncomfortable, and distracting movement during catapult launches. Specifically, officials stated that the nose gear strut moves up and down as an aircraft accelerates to takeoff, which can cause neck and jaw soreness for the pilot because the helmet and oxygen mask are pushed back on the pilot’s face during take-off. This can be a safety risk as the helmet can hit the canopy, possibly resulting in damage, and flight critical symbology on the helmet can become difficult to read during and immediately after launch due to the rotation of the helmet on the pilot’s head. Officials evaluated several options for adjusting the nose gear to alleviate the issue, but determined that none of the options would significantly affect the forces felt by the pilot. Officials subsequently assembled a team to identify a root cause and a redesign. According to officials, adjustments to the catapult system load settings are being considered to address this issue, and a design change to the aircraft may not be required. But flight testing of the proposed changes is required to confirm this solution.

Insulation on coolant tubes: During maintenance on an aircraft in 2016, officials found that insulation around coolant tubes within the aircraft’s fuel system were cracking and contaminating the fuel lines. According to officials, the problem was a result of a supplier using the incorrect material for insulation. The faulty insulation was installed on 57 aircraft— including the entire Air Force initial operational capability fleet—which were prohibited from flight until the insulation was removed. Officials determined that the insulation would not need to be replaced as the aircraft meets specifications without it. Officials are considering removing the insulation from the tubes across the rest of the aircraft going forward. As of January 2017, all of the fielded aircraft have been repaired and returned to flight.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: Travis Masters (Assistant Director), Emily Bond, Raj Chitikila, Kristine Hassinger, Karen Richey, Jillena Roberts, Megan Setser, Hai Tran, and Robin Wilson.

Related GAO Products

F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016.

F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016.

F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016.

F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015.

F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014.

F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014.

F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014.

F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013.

F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013.

F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013.

Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012.

Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-12-400SP. Washington, D.C.: March 29, 2012.

Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012.

Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.

Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.