F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments
Published: 2017-04-24T00:00:00

Fast Facts:
For years we've reported on the F-35 program's cost and schedule overruns, knowledge gaps, and performance issues. Although DOD has taken a number of steps to address these concerns, the department continues to struggle to keep the development costs for the baseline aircraft in check—raising questions about DOD's plan to make significant investments in developing new F-35 capabilities.
Our recommendation? Before committing dollars to future F-35 capabilities, DOD should complete development of the baseline aircraft.

Highlights:
section_title: Why GAO Did This Study
paragraphs: The F-35 Joint Strike Fighter is DOD's most expensive and ambitious acquisition program. Acquisition costs alone are estimated at nearly $400 billion, and beginning in 2022, DOD expects to spend more than $14 billion a year on average for a decade.
The National Defense Authorization Act for Fiscal Year 2015 included a provision for GAO to review the F-35 acquisition program annually until the program reaches full-rate production. This, GAO's second report in response to that mandate, assesses, among other objectives, (1) progress of remaining program development and testing and (2) proposed future plans for acquisition investments. To conduct this work, GAO reviewed and analyzed management reports and historical test data; discussed key aspects of F-35 development with program management and contractor officials; and compared acquisition plans to DOD policy and GAO acquisition best practices.
section_title: What GAO Found
paragraphs: Cascading F-35 testing delays could cost the Department of Defense (DOD) over a billion dollars more than currently budgeted to complete development of the F-35 baseline program. Because of problems with the mission systems software, known as Block 3F, program officials optimistically estimate that the program will need an additional 5 months to complete developmental testing. According to best practices, credible estimates are rooted in historical data. The program's projections are based on anticipated test point achievements and not historical data. GAO's analysis—based on historical F-35 flight test data—indicates that developmental testing could take an additional 12 months (see table below). These delays could affect the start of the F-35's initial operational test and evaluation, postpone the Navy's initial operational capability, and delay the program's full rate production decision, currently planned for April 2019.
Program officials estimate that a delay of 5 months will contribute to a total increase of $532 million to complete development. The longer delay estimated by GAO will likely contribute to an increase of more than $1.7 billion, approximately $1.3 billion of which will be needed in fiscal year 2018.
Meanwhile, program officials project the program will need over $1.2 billion in fiscal year 2018 to start two efforts. First, DOD expects it will need over $600 million for follow-on modernization (known as Block 4). F-35 program officials plan to release a request for Block 4 development proposals nearly 1 year before GAO estimates that Block 3F—the last block of software for the F-35 baseline program—developmental testing will be completed. DOD policy and GAO best practices state that requirements should be approved and a sound business case formed before requesting development proposals from contractors. Until Block 3F testing is complete, DOD will not have the knowledge it needs to present a sound business case for Block 4. Second, the program may ask Congress for more than $650 million in fiscal year 2018 to procure economic order quantities—bulk quantities. However, as of January 2017 the details of this plan were unclear because DOD's 2018 budget was not final and negotiations with the contractors were ongoing. According to internal controls, agencies should communicate with Congress, otherwise it may not have the information it needs to make a fully informed budget decision for fiscal year 2018. Completing Block 3F development is essential for a sound business case and warrants funding priority over Block 4 and economic order quantities at this time.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD use historical data to reassess the cost of completing development of Block 3F, complete Block 3F testing before soliciting contractor proposals for Block 4 development, and identify for Congress the cost and benefits associated with procuring economic order quantities of parts. DOD did not concur with the first two recommendations and partially concurred with the third while outlining actions to address it. GAO continues to believe its recommendations are valid, as discussed in the report.

section_title: Letter
paragraphs: With estimated acquisition costs of nearly $400 billion, the F-35 Lightning  II—also known as the Joint Strike Fighter—is the Department of  Defense’s (DOD) most costly and ambitious acquisition program. Through  this program, DOD is developing and fielding a family of next generation  strike fighter aircraft that integrate low observable (stealth) technologies  with advanced sensors and computer networking capabilities for the  United States Air Force, Marine Corps, and Navy as well as eight  international partners. The F-35 family comprises the F-35A conventional  takeoff and landing variant, the F-35B short takeoff and vertical landing  variant, and the F-35C carrier-suitable variant. The program is  approaching the end of development and will begin increasing production  rates significantly over the next few years. According to current  projections, the U.S. portion of the program will require acquisition  funding of $54 billion over the next 5 years as it approaches full-rate  production. The program is expected to require nearly $276 billion, or  about $12 billion a year on average, through 2038 to complete  development and procure a total of 2,457 aircraft. In addition, DOD  estimates indicate that the F-35 fleet could cost over $1 trillion to operate  and support over its lifetime.
We have reported on F-35 issues for many years. Over time, we have  reported significant cost, schedule, and performance problems and have  made numerous recommendations for improvement. DOD has taken  action to address many of our recommendations to varying degrees. See  appendix I for a matrix of prior GAO reports, recommendations, and DOD  actions. In addition, a list of related GAO products is included at the end  of the report.
The National Defense Authorization Act for Fiscal Year 2015 included a  provision for GAO to review the F-35 acquisition program annually until  the program reaches full-rate production. This is the second report under  that provision. In this report, we assess (1) progress with remaining  program development and testing; (2) DOD’s proposed future plans for  new F-35 acquisition investments; and (3) manufacturing progress,  including supply chain performance.
To conduct our work, we reviewed and analyzed cost performance  reports, test data and results, program briefings, and internal DOD  program analyses. We collected data on and discussed key aspects of F- 35 development progress, including flight testing, with program  management and contractor officials as well as DOD test officials and  program test pilots. We analyzed historical test point execution rates and  cost performance reports to determine the program’s average monthly  progress and to identify the program’s average monthly costs over a 6- month span ending in September 2016. We used those averages along  with the program’s remaining flight test points to calculate the amount of  time and money the program will likely need to finish development. To  assess the reliability of the test and cost data, we reviewed the supporting  documentation and discussed the development of the data with DOD  officials instrumental in producing them. To assess DOD’s proposed  plans for future F-35 investments, we discussed initiatives with program  and contractor officials. We analyzed DOD’s fiscal year 2017 budget  request to identify costs and compared the acquisition plans for these  initiatives to DOD policy and GAO acquisition best practices. We also  collected and analyzed production and supply chain performance data  from DOD, Lockheed Martin (the prime aircraft contractor), and Pratt &  Whitney (the prime engine contractor) and assessed the reliability of the  data by reviewing supporting documentation and interviewing  knowledgeable agency officials. We determined that all of the data we  used were sufficiently reliable for the purposes of this report. Appendix II  contains a detailed description of our scope and methodology.
We conducted this performance audit from June 2016 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: As we have previously reported, DOD began the F-35 acquisition  program in October 2001 without adequate knowledge about the aircraft’s  critical technologies or design. In addition, DOD’s acquisition strategy  called for high levels of concurrency or overlap among development,  testing, and production. In our prior work, we have identified the lack of  adequate knowledge and high levels of concurrency as major drivers of  the significant cost and schedule growth as well as performance shortfalls  that the program has experienced since 2001. The program has been  restructured three times since it began: first in December 2003, again in  March 2007, and most recently in March 2012. The most recent  restructuring was initiated in early 2010 when the program’s unit cost  estimates exceeded critical thresholds established by statute—a condition  known as a Nunn-McCurdy breach. DOD subsequently certified to  Congress in June 2010 that the program was essential to national  security and needed to continue. DOD then began efforts to significantly  restructure the program and establish a new acquisition program  baseline. These restructuring efforts continued through 2011 and into  2012, during which time the department increased the program’s cost  estimates and extended its testing and delivery schedules. Since then  costs have remained relatively stable. Table 1 shows the cost, quantity,  and schedule changes from the initial program baseline and the relative  stability since the new baseline was established.
As the program has been restructured, DOD has also reduced near-term  aircraft procurement quantities. From 2001 and through 2007, DOD  deferred the procurement of 931 aircraft into the future, and then again  from 2007 and through 2012, DOD deferred another 450 aircraft. Figure 1  shows how planned quantities in the near term steadily declined over  time.
The F-35 is DOD’s most costly acquisition program, and over the last  several years we have reported on the affordability challenges facing the  program. As we reported in April 2016, the estimated total acquisition cost  for the F-35 program was $379 billion, and the program would require an  average of $12 billion per year from 2016 through 2038. The program  expects to reach peak production rates for U.S. aircraft in 2022, at which  point DOD expects to spend more than $14 billion a year on average for a  decade (see fig. 2). Given these significant acquisition costs, we found  that DOD would likely face affordability challenges as the F-35 program  competes with other large acquisition programs, including the B-21  bomber, KC-46A tanker, and Ohio Class submarine replacement. In  addition, in September 2014, we reported that DOD’s F-35 sustainment  strategy may not be affordable.
Through 2016, DOD had awarded contracts for production of 9 lots of F- 35 aircraft, totaling 285 aircraft (217 aircraft for the U.S. and 68 aircraft for  international partners or foreign military sales). At the time of this report,  the contract for lot 10 had not been signed.
In 2013, the Departments of the Navy and the Air Force issued a joint  report to the congressional defense committees providing that the Marine  Corps and Air Force would field initial operating capabilities in 2015 and  2016, respectively, with aircraft that had limited warfighting capabilities.  The Navy did not plan to field its initial operating capability until 2018,  after the F-35’s full warfighting capabilities had been developed and  tested. These dates represented a delay of 5 to 6 years from the  program’s initial baseline. As planned, the Marine Corps and Air Force  declared initial operational capability (IOC) in July 2015 and August 2016,  respectively.
section_title: Continuing Problems with Flight Testing of Software Capabilities Will Delay the Completion of Development and Increase Development Costs
paragraphs: DOD will need more time and money than expected to complete the  remaining 10 percent of the F-35 development program. DOD has  experienced delays in testing the software and systems that provide  warfighting capabilities, known as mission systems, largely because the  software has been delivered late to be tested and once delivered has not  worked as expected. Program officials have had to regularly divert  resources from developing and testing of more advanced software  capabilities to address unanticipated problems with prior software  versions. These problems have compounded over time, and this past  year was no exception. DOD began testing the final block of software— known as block 3F—later than expected, experienced unanticipated  problems with the software’s performance, and thus did not complete all  mission systems testing it had planned for 2016. As a result, the F-35  program office has noted that more time and money will be needed to  complete development. The amount of time and money could vary  significantly depending on the program’s ability to complete  developmental and operational testing. We estimate that developmental  testing could be delayed as much as 12 months, thus delaying the start of  initial operational testing, and total development costs could increase by  nearly $1.7 billion. In addition, the Navy’s IOC and the program’s full-rate  production decision could also be delayed.
subsections: section_title: Problems with Mission Systems Software Continue to Cause Delays in Flight Testing
paragraphs: DOD continues to experience delays in F-35 mission systems testing.  Although mission systems testing is about 80 percent complete, the  complexity of developing and testing mission systems has been  troublesome. For the F-35 program, DOD is developing and fielding  mission systems capabilities in software blocks: (1) Block 1, (2) Block 2A,  (3) Block 2B, (4) Block 3i, and (5) Block 3F. Each subsequent block builds  on the capabilities of the preceding block. Over the last few years,  program officials have had to divert resources—personnel and  infrastructure—from developing and testing of more advanced software  blocks to address unanticipated problems with prior software blocks. Over  time, this practice has resulted in compounding delays in mission systems  testing. Blocks 1 through 3i are now complete, and the program is  currently focused on developing and testing Block 3F, the final software  block in the current development program. Figure 3 illustrates the mission  systems software blocks being developed for the program, the  percentage of test points completed by block, and the build-up to full  warfighting capability with Block 3F.
Program officials spent some of 2016 addressing problems with Block 3i  mission systems unexpectedly shutting down and restarting—an issue  known as instability—which delayed Block 3F testing. In early 2016,  officials were developing and testing Block 3i concurrently with Block 3F.  In order to ensure that the Block 3i instability was addressed in time for  the Air Force’s planned IOC in August 2016, officials diverted resources  from Block 3F.That decision delayed subsequent testing that had been  planned for Block 3F. Further delays resulted from the discovery of  instability and functionality problems with Block 3F. To mitigate some  schedule delays, program officials implemented a new process to  introduce software updates quicker than normal. Although the quick  software releases helped to ensure that testing continued, the final  planned version of Block 3F, which was originally planned to be released  to testing in February 2016, was not released until late November 2016,  nearly a 10-month delay. As a result, program officials have identified the  need for additional time to complete development. Program officials now  project that developmental testing, which was expected to be completed  in May, will conclude in October 2017, 5 months later than planned.
However, based on our analysis, the program’s projection is optimistic as  it does not reflect historical F-35 test data. Program officials believe that  going forward they may be able to devote more resources to mission  systems testing, which could lead to higher test point completion rates  than they have achieved in the past. According to GAO best practices,  credible schedule estimates are rooted in historical data. As of November  2016, program officials estimated that the program will need to complete  as much as an average of 384 mission systems test points per month in  order to finish flight testing by October 2017—a rate that the program has  rarely achieved before. Our analysis of historical test point data as of  December 2016 indicates that the average test point execution rates are  much lower, at 220 mission systems test points per month. In addition,  historical averages suggest that test point growth—additions to the overall  test points from discovery in flight testing—is much higher than program  officials assume, while estimated deletions—test points that are  considered no longer required—are lower than assumed. Using the  historical F-35 averages, we project that developmental testing may not  be completed until May 2018, a 12-month delay from the program’s  current plan. Table 2 provides a comparison of the assumptions used to  determine delays in developmental testing.
Our estimation of delays in completing developmental testing does not  include the time it may take to address the significant number of existing  deficiencies. The Marine Corps and Air Force declared IOC with limited  capability and with several deficiencies. As of October 2016, the program  had more than 1,200 open deficiencies, and senior program and test  officials deemed 276 of those critical or of significant concern to the  military services. Several of the critical deficiencies are related to the  aircraft’s communications, data sharing, and target tracking capabilities.  Although the final planned version of Block 3F software was released to  flight testing in November 2016 and contained all 332 planned warfighting  capabilities, not all of those capabilities worked as intended. In  accordance with program plans, it was the first time some of the Block 3F  capabilities had been tested. According to a recent report by the Director,  Operational Test and Evaluation (DOT&E), fixes for less than half of the  276 deficiencies were included in the final planned version of Block 3F  software. Prime contractor officials stated that additional software  releases will likely be required to address deficiencies identified during  the testing of the final planned version of Block 3F software, but they do  not yet know how many releases will ultimately be needed.
section_title: Delays in Developmental Testing Will Likely Affect Plans for Initial Operational Testing, Navy Operational Capability Dates, and Full-Rate Production
paragraphs: Delays in developmental testing will likely drive delays in current plans to  start F-35 initial operational test and evaluation. Program officials have  noted that according to their calculations developmental testing will end in  October 2017 and initial operational testing will begin in February 2018.  However, DOT&E officials, who approve operational test plans, anticipate  that the program will more likely start operational testing in late 2018 or  early 2019, at the earliest. Figure 4 provides an illustration of the current  program schedule and DOT&E’s projected delays.
DOT&E’s estimate for the start of initial operational testing is based on  the office’s projection that developmental testing will end in July 2018 and  that retrofits needed to prepare the aircraft for operational testing will not  be completed until late 2018 at the earliest. There are 23 aircraft—many  of which are early production aircraft—that require a total of 155 retrofits  before they will be ready to begin operational testing. As of January 2017,  20 of those retrofits were not yet under contract, and program officials  anticipated some retrofits would be completed in late 2018.
To mitigate possible schedule delays, program officials are considering a  phased start to operational testing. However, current program test plans  require training and preparation activities before initial operational test  and evaluation begins. Those activities, as outlined in the test plan, are  expected to take approximately 6 months. Changes to this approach  would require approval from DOT&E. According to DOT&E officials,  however, the program has not yet provided any detailed strategy for  implementing a new approach or identified a time frame for revising the  test plan.
Significant delays in initial operational testing will likely affect two other  upcoming program decisions: (1) the Navy’s decision to declare IOC and  (2) DOD’s decision to begin full-rate production. In a 2015 report to the  congressional defense committees, the Under Secretary of Defense for  Acquisition, Technology and Logistics stated that the Navy’s IOC  declaration is on track for February 2019 pending completion of initial  operational test and evaluation. If initial operational testing does not  begin until February 2019 as the DOT&E predicts, the Navy may need to  consider postponing its IOC date. Likewise, DOD’s full-rate production  decision, currently planned for April 2019, may have to be delayed.  According to statute, a program may not proceed beyond low-rate initial  production into full-rate production until initial operational test and  evaluation is completed and DOT&E has submitted to the Secretary of  Defense and the congressional defense committees a report that  analyzes the results of operational testing. If testing does not begin until  February 2019 and takes 1 year, as expected, DOD will not have the  report in time to support a full-rate production decision by April 2019.
section_title: Delays in Development Will Increase Program Costs and Add to Concurrency
paragraphs: The current delays in F-35 developmental testing will also result in  increased development costs. Based on the program office’s estimate of  a 5-month delay in developmental testing, the F-35 program will need an  additional $532 million to complete the development contract. According  to GAO best practices, credible cost estimates are also rooted in  historical data. Using historical contractor cost data from April 2016 to  September 2016, we calculated the average monthly cost associated with  the development contract. If developmental testing is delayed 12 months,  as we estimate, and operational testing is not completed until 2020, as  projected by DOT&E, then we estimate that the program could need more  than an additional $1.7 billion to complete the F-35 development contract.  Similarly, the Cost Assessment and Program Evaluation office within the  Office of the Secretary of Defense has estimated that the program will  likely need more than $1.1 billion to complete the development contract.  In these estimates, the majority of the additional funding would be needed  in fiscal year 2018. Specifically, program officials believe that an  additional $353.8 million may be needed in fiscal year 2018, while we  estimate that they could need more than three times that amount— approximately $1.3 billion—as illustrated in figure 5.
The program plans to fund their estimated development program deficit  through several means. For example, although the program office 2018  preliminary budget projection reflected a reduction of $81 million in  development funding over the next few years, as compared to DOD’s  fiscal year 2017 budget request, program officials expect DOD to restore  this reduction in its official fiscal year 2018 budget request. In addition,  program officials plans to increase the budget request, as compared to  their fiscal year 2017 budget request, for development funding in fiscal  years 2018, 2019, and 2020 by $451 million and likewise reduce their  budget request for procurement funding over those years. To make up for  the reduction in requested procurement funding, the program plans to  reprogram available procurement funds appropriated in prior fiscal  years. Any additional funding beyond $451 million would likely have to  come from some other source. Figure 5 compares DOD’s and our  estimates for development funding needs from fiscal years 2018 through  2021.
As developmental testing is delayed and DOD procures more aircraft  every year, concurrency costs—the costs of retrofitting delivered  aircraft—increase. For example, from 2015 to 2016, the program  experienced a $70 million increase in concurrency costs. This increase  was partially driven by the identification of new technical issues found  during flight testing that were not previously forecasted, including  problems with the F-35C outer-wing structure and F-35B landing gear.  Problems such as these have to be fixed on aircraft that have already  been procured. Thus far, DOD has procured 285 aircraft and has  experienced a total of $1.77 billion in concurrency costs. Although testing  is mostly complete, any additional delays will likely result in delays in the  incorporation of known fixes, which would increase the number of aircraft  that will require retrofits and rework and further increase concurrency  costs as more aircraft are procured. According to program officials, most  of the retrofits going forward are likely to be software related and thus  less costly. However, according to DOD’s current plan, 498 aircraft will be  procured by the time initial operational testing is complete. If the  completion of operational testing is delayed to 2020, as DOT&E predicts,  the number of procured aircraft will increase to 584 as currently planned,  making 86 additional aircraft subject to any required retrofits or rework.
section_title: DOD’s Planned Investments in Modernization and Economic Order Purchase May Be Premature
paragraphs: In fiscal year 2018, F-35 program officials expect to invest more than $1.2  billion to start two efforts while simultaneously facing significant shortfalls  in completing the F-35 baseline development program, as discussed  above. Specifically, DOD and program officials project that in fiscal year  2018 the program will need over $600 million to begin development of  follow-on modernization of the F-35 and more than $650 million to  procure economic order quantities (EOQ) of parts to achieve cost savings  during procurement. Contracting for EOQ generally refers to the  purchase of parts in larger more economically efficient quantities to  minimize the cost of these items. DOD officials emphasized that the  specific amount of funding needed for these investments could change as  the department finalizes its fiscal year 2018 budget request. Regardless,  these investments may be premature. Early Block 4 requirements, which  represent new capabilities beyond the original requirements, may not be  fully informed before DOD plans to solicit proposals from contractors for  how they might meet the government’s requirements—a process known  as request for proposal (RFP). According to DOD policy, the  Development RFP Release Decision Point is the point at which a solid  business case is formed for a new development program. Until Block 3F  testing is complete, DOD will not have the knowledge it needs to develop  and present an executable business case for Block 4, with reliable cost  and funding estimates.
subsections: section_title: Follow-on Modernization May Be Too Early
paragraphs: Due to evolving threats and changing warfighting environments, program  officials project that the program will need over $600 million in fiscal year  2018 to award a contract to begin developing new F-35 capabilities, an  effort referred to as follow-on modernization. However, the requirements  for the first increment of that effort, known as Block 4, have not been  finalized. Block 4 is expected to be developed and delivered in four  phases—currently referred to as 4.1, 4.2, 4.3, and 4.4. Program officials  expect phases 4.1 and 4.3 to be primarily software updates, while 4.2 and  4.4 consist of more significant hardware changes. The program has  drafted a set of preliminary requirements for Block 4 that focused on the  top-level capabilities needed in phases 4.1 and 4.2, but the requirements  for the final two phases have not been fully defined. In addition, as of  January 2017, these requirements had not been approved by the Joint  Requirements Oversight Council.
Delays in developmental testing of Block 3F are also likely to affect Block  4 requirements. DOD policy states that requirements are to be approved  before a program reaches the Development RFP Decision Point in the  acquisition process. GAO best practices emphasize the importance of  matching requirements and resources in a business case before a  development program begins. For DOD, the Development RFP Release  Decision Point is the point at which plans for the program must be most  carefully reviewed to ensure that all requirements have been approved,  risks are understood and under control, the program plan is sound, and  the program will be affordable and executable. Currently, F-35 program  officials plan to release the RFP for Block 4.1 development in the third  quarter of fiscal year 2017, nearly 1 year before we estimate Block 3F  developmental testing will be completed. Program officials have stated  that Block 3F is the foundation for Block 4, but continuing delays in Block  3F testing make it difficult to fully understand Block 3F functionality and its  effect on early Block 4 capabilities. If new deficiencies are identified  during the remainder of Block 3F testing, the need for new technologies  may arise, and DOD may need to review Block 4 requirements again  before approving them.
section_title: Costs and Benefits of Economic Order Quantity Purchases Are Not Clear
paragraphs: In April 2016, we reported that the F-35 program office was considering  what it referred to as a block buy contracting approach that we noted had  some potential economic benefits but could limit congressional funding  flexibility. The program office has since changed its strategy to consist  of contracts for EOQ of 2 years’ worth of aircraft parts followed by a  separate annual contract for procurement of lot 12 aircraft with annual  options for lots 13 and 14 aircraft. Each of these options would be  negotiated separately, similar to how DOD currently negotiates contracts.
As of January 2017, details of the program office’s EOQ approach were  still in flux. In 2015, the program office contracted with RAND Corporation  to conduct a study of the potential cost savings associated with several  EOQ approaches. According to the results of that study, in order for the  government to get the greatest benefit, the aircraft and engine contractors  would need to take on risk by investing in EOQ on behalf of the  department in fiscal year 2017. Program officials envision that under this  arrangement the contractors would be repaid by DOD at a later date.  However, as of January 2017, contractors stated they were still  negotiating the terms of this arrangement; therefore, the specific costs  and benefits remained uncertain. Despite this uncertainty, the program  office plans to seek congressional approval to make EOQ purchases and  expects to need more than $650 million for that purpose in fiscal year  2018. Program officials believe that this upfront investment would result in  a significant savings over the next few years for the U.S. services.  However, given the uncertainties around the level of contractor  investment, it is not clear whether an investment of more than $650  million, if that is the final amount DOD requests in fiscal year 2018, will be  enough to yield significant savings. Regardless, with cost growth and  schedule delays facing the F-35 baseline development program, it is  unclear whether DOD can afford to fund this effort at this time. According  to internal control standards, agencies should communicate with external  stakeholders, such as Congress. With a potential investment of this  size, particularly in an uncertain budget environment, it is important that  program officials finalize the details of this approach before asking for  congressional approval and provide Congress with a clear understanding  of the associated costs to ensure that funding decisions are fully  informed.
section_title: Aircraft Manufacturing and Reliability Are Improving in Some Areas
paragraphs: The F-35 airframe and engine contractors continue to report improved  manufacturing efficiency, and program data indicate that reliability and  maintainability are improving in some areas. Over the last 5 years, the  number of U.S. aircraft produced and delivered by Lockheed Martin has  increased, and manufacturing efficiency and quality have improved over  time. Similarly, manufacturing efficiency and quality metrics are improving  for Pratt & Whitney. Although some engine aircraft reliability and  maintainability metrics are not meeting program expectations, there has  been progress in some areas, and there is still time for further  improvements.
subsections: section_title: Airframe and Engine Manufacturing Efficiency Is Improving, but Supply Chain Challenges Remain
paragraphs: Overall the airframe manufacturer, Lockheed Martin, is improving  efficiency and product quality. Over the last 5 years, the number of  aircraft produced and delivered by Lockheed Martin has increased from  29 aircraft in 2012 to 46 aircraft in 2016. Since 2011, a total of 200  production aircraft have been delivered to DOD and international  partners, 46 of which were delivered in 2016. As of January 2017, 142  aircraft were in production, worldwide. As more aircraft are delivered,  the number of labor hours needed to manufacture each aircraft declines.  Labor hours decreased from 2015 to 2016, indicating production maturity.  In addition, instances of production line work done out of sequence  remains relatively low, with the exception of an increase at the end of  2016 due to technical issues, such as repairing coolant tube insulation  (see app. III). Further, the number of quality defects and total hours spent  on scrap, rework, and repair declined in 2016.
Although data indicate that airframe manufacturing efficiency and quality  continue to improve, supply chain challenges remain. Some suppliers are  delivering late and non-conforming parts, resulting in production line  inefficiencies and workarounds. For example, in 2016, Lockheed Martin  originally planned to deliver 53 aircraft, but quality issues with insulation  on the coolant tubes in the fuel tanks resulted in the contractor delivering  46 aircraft. According to Lockheed Martin officials, late deliveries of parts  are largely due to late contract awards and supply base capacity. While  supplier performance is generally improving, it is important for suppliers to  be prepared for both production and sustainment support going forward.  Inefficiencies, such as conducting production line work out of sequence,  could be exacerbated if late delivery of parts continues as production  more than doubles over the next 5 years.
The engine manufacturer, Pratt & Whitney, is also improving efficiency.  As of October 2016, Pratt & Whitney had delivered 279 engines. The  labor hours required to assemble an F-35 engine decreased quickly and  has remained relatively steady since around the 70th engine produced,  and little additional efficiency is expected to be gained. Other Pratt &  Whitney manufacturing metrics indicate that production efficiency and  quality are improving. Scrap, rework, and repair costs were reduced from  2.22 percent in 2015 to 1.8 percent in 2016. We previously reported that  according to Pratt & Whitney officials, moving from a hollow blade design  to a solid blade would reduce scrap and rework costs because it is easier  to produce. However, Pratt & Whitney experienced unanticipated  problems with cracking in the solid blade design. As a result, Pratt &  Whitney is continuing to produce a hollow blade while it further  investigates the difficulty and costs associated with a solid blade design.  Pratt & Whitney’s supply chain continues to make some improvements.  For example, critical parts are being delivered ahead of schedule, and  some are already achieving 2017 rate requirements. To further ensure  that suppliers are capable of handling full-rate production, Pratt & Whitney  is pursuing the potential to have multiple suppliers for some engine parts,  which officials believe will help increase manufacturing capacity within the  supply chain.
section_title: Aircraft Reliability and Maintainability Metrics Have Improved in Some Areas
paragraphs: Although the program has made progress in improving system-level  reliability and maintainability, some metrics continue to fall short of  program expectations in several key areas. For example, as shown in  figure 6, while metrics in most areas were overall trending in the right  direction, the F-35 program office’s internal assessment indicated that as  of August 2016 the F-35 fleet was falling short of reliability and  maintainability expectations in 11 of 21 areas.
Although many of the metrics remain below program expectations, some  of the metrics have shown improvement over the last year, and time  remains for continued improvements. For example, our analysis indicates  that since 2015, the F-35A reliability has improved from 4.3 mean flight  hours between failure attributable to design issues to 5.7 hours, nearly  achieving the goal at system maturity of 6 hours. The F-35A mean flight  hours between maintenance event metric has also improved and is now  meeting program expectations. As of August 2016, the F-35 fleet had only  flown a cumulative total of 63,187 flight hours. The program has time for  further improvement as the ultimate goals for these reliability and  maintainability metrics are to be achieved by full system maturity, or  200,000 cumulative flight hours across the fleet. The program also plans  to improve these metrics through additional design changes.
Engine reliability varied in 2016. In April 2016, we reported that Pratt &  Whitney had implemented a number of design changes that resulted in  significant improvements to one reliability metric: mean flight hours  between failure attributable to design issues. At the time of our report,  contractor data indicated the F-35A and F-35B engines were at about 55  percent and 63 percent, respectively, of where the program expected  them to be. According to contractor data as of September 2016, the  program was unable to achieve a significant increase in reliability over the  last year, which left the F-35A and F-35B engines further below  expectations—at about 43 percent and 41 percent, respectively. Other  reliability metrics such as engine’s impact on aircraft availability, engine  maintenance man-hours, and the time between engine removals are  meeting expectations. On average, from June 2016 through November  2016, the engine affected only about 1.47 percent of the overall aircraft  availability rates, and none of the top 30 drivers affecting aircraft  availability were related to the engine. According to Pratt & Whitney  officials, the F-35 engine requires fewer maintenance man-hours per flight  hour than legacy aircraft, and engines for the F-35A and F-35B are  currently performing better than required for the average number of flight  hours between engine removals. Program and contractor officials  continue to identify ways to further improve reliability through a number of  design changes and expect reliability to continue to improve lot over lot.
section_title: Conclusions
paragraphs: As the F-35 program approaches the end of development, its schedule  and cost estimates are optimistic. The program’s cost and schedule  estimates to complete development are hundreds of millions of dollars  below and several months under other independent estimates, including  our own. If the program experiences schedule delays as we predict, it  could require a total of nearly $1.5 billion in fiscal year 2018 alone.  However, program officials project that the program will only need $576.2  million in fiscal year 2018 to complete baseline development. At the same  time, program officials expect that more than $1.2 billion could be needed  to commit to Block 4 and EOQ in fiscal year 2018. DOD must prioritize  funding for the baseline development program over the program office’s  desire for EOQ and Block 4. If baseline development is not prioritized and  adequately funded, and costs increase as predicted by GAO and others,  then the program will have less recourse for action and development  could be further delayed. In addition, with baseline development still  ongoing the program will not likely have the knowledge it needs to  present a sound business case for soliciting contractor proposals for  Block 4 development in fiscal year 2017. Although Block 4 and EOQ may  be desirable, prioritizing funding for these efforts may not be essential at  this time. Prioritizing funding for baseline development over these two  efforts would ensure that the program has the time and money needed to  properly finish development and thus lay a solid knowledge-based  foundation for future efforts.
section_title: Recommendations for Executive Action
paragraphs: To ensure that DOD adequately prioritizes its resources to finish F-35  baseline development and delivers all of the promised warfighting  capabilities and that Congress is fully informed when making fiscal year  2018 budget decisions, we are making the following three  recommendations to the F-35 program office through the Secretary of  Defense.  1.  Reassess the additional cost and time needed to complete  developmental testing using historical program data.  2.  Delay the issuance of the Block 4 development request for proposals  at least until developmental testing is complete and all associated  capabilities have been verified to work as intended.  3.  Finalize the details of DOD and contractor investments associated  with an EOQ purchase in fiscal year 2018, and submit a report to  Congress with the fiscal year 2018 budget request that clearly  identifies the details, including costs and benefits of the finalized EOQ  approach.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. DOD’s  comments are reprinted in appendix IV and summarized below. DOD also  provided technical comments, which were incorporated as appropriate.
DOD did not concur with our recommendation to reassess the additional  cost and time needed to complete developmental testing using historical  program data. DOD stated that it will continue to assess the assumptions  and decisions made, and communicate any necessary adjustments  relative to both cost and time needed to complete developmental testing.
DOD also stated that it had considered historical data in its assessment  and concluded that developmental testing could extend into February  2018. While this possible slip is noted in our report, it is unclear to us the  extent to which the data underpinning DOD’s assessment reflected the  program’s historical averages. While the program’s analysis that we  examined did reflect test point accomplishment rates that were more  aligned with what the program achieved in 2016 (i.e. around 290 points  per month) those rates were still higher than the historical average. Other  key inputs to that analysis also differed significantly from the program’s  historical averages. For example, program officials assumed only a 42  percent test point growth rate when the program’s historical average test  point growth was 63 percent, and in 2016 alone the test point growth rate  was 115 percent. Several other DOD officials have identified possible  delays beyond February 2018. In a memo sent to Congress in December  2016, the Under Secretary of Defense for Acquisition, Technology and  Logistics stated that developmental testing could go as long as May 2018,  and DOT&E analysis also indicates that developmental testing may not  conclude until mid-2018. We continue to believe that our recommendation  is valid.
DOD also did not concur with our recommendation to delay the issuance  of the Block 4 development request for proposals until developmental  testing is complete. According to DOD, delaying the request for proposals  could unnecessarily delay delivery of needed capabilities to the  warfighters. However, as program officials stated, Block 3F software  establishes the foundation for Block 4. Therefore, continuing delays in  Block 3F testing will likely make it difficult to fully understand Block 3F  functionality and its effect on early Block 4 requirements. If new  deficiencies are identified during the remainder of Block 3F testing, the  need for new technologies may arise, and DOD may need to review Block  4 requirements again before approving them which could lead to  additional delays. Therefore, we continue to believe that our  recommendation is valid.
DOD stated that it partially concurred with our third recommendation to  finalize the details of investments associated with an EOQ purchase in  fiscal year 2018, and submit a report to Congress with the fiscal year  2018 budget request that clearly identifies those details. However, in its  response, the department outlined steps that address it. For example,  DOD stated that it had finalized the details of DOD and contractor  investments associated with an EOQ purchase and will brief Congress on  the details, including costs and benefits of the finalized EOQ approach.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; and the Under Secretary of  Defense for Acquisition, Technology and Logistics. In addition, the report  is available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.
section_title: Appendix I: Prior GAO Report and DOD Actions
section_title: Appendix II: Scope and Methodology
paragraphs: To assess the F-35 program’s remaining development and testing we  interviewed officials from the program office and contractors—Lockheed  Martin and Pratt & Whitney. We obtained and analyzed data on mission  systems test point execution, both planned and accomplished from 2011  through 2016 to calculate historical test point averages per month. We  compared test progress against the total program requirements to  determine the number of test points that were completed and remaining  as of December 2016. We used the average test point rate based on the  historical data to determine the number of months needed to complete  the remaining test points. To identify the program’s average monthly  costs, we analyzed contractor cost performance data from April 2016  through September 2016 to identify average contract costs per month.  Using a 12-month delay and the average contract costs per month, we  calculated the costs to complete developmental testing. In order to  determine costs to complete development, we first determined the  percent change, year to year, in the program office’s development funding  requirement estimate from 2018 to 2021. We then reduced our estimate  using those percentages from 2018 to 2021. We discussed key aspects  of F-35 development progress, including flight testing progress, with  program management and contractor officials as well as DOD test  officials and program test pilots. To assess the reliability of the test and  cost data, we reviewed the supporting documentation and discussed the  development of the data with DOD officials instrumental in producing  them. In addition, we interviewed officials from the F-35 program office,  Lockheed Martin, Pratt & Whitney, and the Director, Operational Test and  Evaluation office to discuss development test plans, achievements, and  test discoveries.
To assess DOD’s proposed plans for future F-35 investments, we  discussed cost and manufacturing efficiency initiatives, such as the  economic order quantities approach, with contractor and program office  officials to understand potential cost savings and plans. To assess the  program’s follow-on modernization plans, we discussed the program’s  plans with program office officials. We reviewed the fiscal year 2017  budget request to identify costs associated with the effort. We also  reviewed and analyzed best practices identified by GAO and reviewed  relevant DOD policies and statutes. We compared the acquisition plans to  these policies and practices.
To assess ongoing manufacturing and supply chain performance, we  obtained and analyzed data related to aircraft delivery rates and work  performance data from January 2016 to December 2016. These data  were compared to program objectives identified in these areas and used  to identify trends. We reviewed data and briefings provided by the  program office, Lockheed Martin, Pratt & Whitney, and the Defense  Contract Management Agency in order to identify issues in manufacturing  processes. We discussed reasons for delivery delays and plans for  improvement with Lockheed Martin and Pratt & Whitney. We collected  and analyzed data related to aircraft quality through December 2016. We  collected and analyzed supply chain performance data and discussed  steps taken to improve quality and deliveries with Lockheed Martin and  Pratt & Whitney. We also analyzed reliability and maintainability data and  discussed these issues with program and contractor officials.
We assessed the reliability of DOD and contractor data by reviewing  existing information about the data and interviewing agency officials  knowledgeable about the data. We determined that the data were  sufficiently reliable for the purposes of this report.
We conducted this performance audit from June 2016 to April 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix III: Status of F-35 Technical Risks
paragraphs: As developmental testing nears completion, the F-35 program continues  to address technical risks. The program has incorporated design changes  that appear to have mitigated several of the technical risks that we have  highlighted in prior reports, including problems with the arresting hook  system and bulkhead cracks on the F-35B. However, over the past year,  the program continued to address risks with the Helmet Mounted Display,  Autonomic Logistics Information System (ALIS), the ejection seat and  engine seal that we have identified in the past. The program also  identified new risks with the F-35C wing structure and catapult launches,  and coolant tube insulation. The status of the Department of Defense’s  (DOD) efforts to address these issues is as follows:  Helmet Mounted Display: A new helmet intended to address shortfalls in  night vision capability, among other things, was developed and delivered  to the program in 2015. Developmental testing of the new helmet is  mostly complete, and officials believe that issues such as latency and  jitter have been addressed. Green glow, although improved, continues to  add workload for the pilots when landing at sea. Officials believe that they  have done as much as they can to fix the green glow problems with the  hardware currently available.
ALIS: ALIS continues to lack required capabilities; for instance, engine  parts information is not included in the current version of ALIS, although it  is expected to be completed in the spring of 2017. In 2016, officials began  testing ALIS in an operational environment which has led to some  improvements. However, capabilities, including the prognostics health  management downlink, have been deferred to follow-on modernization. In  2016, officials acknowledged compounding development delays and  restructured the development schedule for ALIS. The new schedule  shows that some capabilities that were planned in the earlier versions of  ALIS will now be deferred to later versions. In April 2016, we reported that  F-35 pilots and maintainers identified potential functionality risks to ALIS  and that DOD lacked a plan to address these risks as key milestone  dates approached, which could result in operational and schedule risks.
Engine seal: Officials have identified a design change to address the  technical problem that resulted in an engine fire in June 2014. This design  change was validated and incorporated into production in 2015. Engine  contractor officials identified 194 engines that needed to be retrofitted,  and as of October 2016, 189 of those retrofits had been completed. The  engine contractor, Pratt & Whitney, is paying for these retrofits.
Ejection seat: In 2015, officials discovered that pilots who weigh less than  136 pounds could possibly suffer neck injuries during ejection. Officials  stated that the risk of injury is due to the over-rotation of the ejection seat  in combination with the thrust from the parachute deployment during  ejection. Officials noted that although the problem was discovered during  testing of the new Helmet Mounted Display, the helmet’s weight was not  the root cause. The program has explored a number of solutions to  ensure pilot safety including installing a switch for light-weight pilots that  would slow the release of the parachute deployment, installing a head  support panel that would reduce head movement, and reducing the  weight of the helmet. The final design completed qualification testing in  2016 and is expected to be incorporated into production lot 10. The cost  of these changes has not yet been determined.
F-35C outer-wings: In 2016, officials identified structural issues on the F- 35C outer-wing when carrying an AIM-9X missile. In order to resume the  test program, officials identified a design change to include strengthening  the wings’ material that was incorporated onto a test aircraft. Officials  expect to incorporate retrofits to delivered aircraft by 2019 and will  incorporate changes into production in lot 10.
F-35C catapult launches: In 2016, officials identified issues with violent,  uncomfortable, and distracting movement during catapult launches.  Specifically, officials stated that the nose gear strut moves up and down  as an aircraft accelerates to takeoff, which can cause neck and jaw  soreness for the pilot because the helmet and oxygen mask are pushed  back on the pilot’s face during take-off. This can be a safety risk as the  helmet can hit the canopy, possibly resulting in damage, and flight critical  symbology on the helmet can become difficult to read during and  immediately after launch due to the rotation of the helmet on the pilot’s  head. Officials evaluated several options for adjusting the nose gear to  alleviate the issue, but determined that none of the options would  significantly affect the forces felt by the pilot. Officials subsequently  assembled a team to identify a root cause and a redesign. According to  officials, adjustments to the catapult system load settings are being  considered to address this issue, and a design change to the aircraft may  not be required. But flight testing of the proposed changes is required to  confirm this solution.
Insulation on coolant tubes: During maintenance on an aircraft in 2016,  officials found that insulation around coolant tubes within the aircraft’s fuel  system were cracking and contaminating the fuel lines. According to  officials, the problem was a result of a supplier using the incorrect  material for insulation. The faulty insulation was installed on 57 aircraft— including the entire Air Force initial operational capability fleet—which  were prohibited from flight until the insulation was removed. Officials  determined that the insulation would not need to be replaced as the  aircraft meets specifications without it. Officials are considering removing  the insulation from the tubes across the rest of the aircraft going forward.  As of January 2017, all of the fielded aircraft have been repaired and  returned to flight.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff members  made key contributions to this report: Travis Masters (Assistant Director),  Emily Bond, Raj Chitikila, Kristine Hassinger, Karen Richey, Jillena  Roberts, Megan Setser, Hai Tran, and Robin Wilson.
section_title: Related GAO Products
paragraphs: F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans  to Begin Development of New Capabilities. GAO-16-390. Washington,  D.C.: April 14, 2016.
F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its  Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016.
F-35 Joint Strike Fighter: Preliminary Observations on Program Progress.  GAO-16-489T. Washington, D.C.: March 23, 2016.
F-35 Joint Strike Fighter: Assessment Needed to Address Affordability  Challenges. GAO-15-364. Washington, D.C.: April 14, 2015.
F-35 Sustainment: Need for Affordable Strategy, Greater Attention to  Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.:  September 23, 2014.
F-35 Joint Strike Fighter: Slower Than Expected Progress in Software  Testing May Limit Initial Warfighting Capabilities. GAO-14-468T.  Washington, D.C.: March 26, 2014.
F-35 Joint Strike Fighter: Problems Completing Software Testing May  Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322.  Washington, D.C.: March 24, 2014.
F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but  Affordability Challenges and Other Risks Remain. GAO-13-690T.  Washington, D.C.: June 19, 2013.
F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but  Affordability Challenges and Other Risks Remain. GAO-13-500T.  Washington, D.C.: April 17, 2013.
F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term  Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March  11, 2013.
Fighter Aircraft: Better Cost Estimates Needed for Extending the Service  Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.:  November 15, 2012.
Joint Strike Fighter: DOD Actions Needed to Further Enhance  Restructuring and Address Affordability Risks. GAO-12-437. Washington,  D.C.: June 14, 2012.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-12-400SP. Washington, D.C.: March 29, 2012.
Joint Strike Fighter: Restructuring Added Resources and Reduced Risk,  but Concurrency Is Still a Major Concern. GAO-12-525T. Washington,  D.C.: March 20, 2012.
Joint Strike Fighter: Implications of Program Restructuring and Other  Recent Developments on Key Aspects of DOD’s Prior Alternate Engine  Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19,  2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.
Joint Strike Fighter: Restructuring Should Improve Outcomes, but  Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.:  March 15, 2011.