F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates
Published: 2014-09-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The F-35 Lightning II is intended to replace a variety of existing aircraft in the Air Force, Navy, and Marine Corps, while providing the most supportable, technologically advanced, lethal, and survivable aircraft to date. The F-35 is DOD's most expensive weapon system, with estimated sustainment costs of about $1 trillion. With the military services planning for the ability to deploy and maintain the F-35 within 4 years, DOD is working to develop a sustainment strategy that will be both affordable and executable for the program's life cycle.
GAO was mandated to review DOD's F-35 sustainment planning efforts. This report addresses the extent to which DOD has (1) developed an F-35 sustainment strategy and addressed potential risks related to affordability and operational readiness and (2) developed a reliable O&S cost estimate for the program's life cycle. GAO analyzed documented plans and cost estimates and interviewed DOD and contractor officials.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) currently has or is developing several plans and analyses that will make up its overall F-35 sustainment strategy, which is expected to be complete in fiscal year 2019. The annual F-35 operating and support (O&S) costs are estimated to be considerably higher than the combined annual costs of several legacy aircraft (see fig.). DOD has begun some cost-savings efforts and established sustainment affordability targets for the F-35 program, but DOD did not use the military services' budgets to set these targets. Therefore, these targets may not be representative of what the services can afford and do not provide a clear benchmark for DOD's cost-savings efforts. In addition, DOD has not fully addressed several issues that have an effect on affordability and operational readiness, including aircraft reliability and technical-data rights, which could affect the development of the sustainment strategy.
It is unclear whether DOD's O&S cost estimates for the F-35 program reflect the most likely costs that the F-35 program will incur. DOD has two primary F-35 O&S estimates that each total around $1 trillion over a 56-year life cycle. These cost estimates are comprehensive in that they include all DOD-required program elements and are organized according to a standard O&S cost-estimating structure; however, weaknesses exist with respect to a few of the assumptions, and the estimates did not include all analyses necessary to make them fully reliable. For example, the estimates did not use reasonable fuel burn rate assumptions that reflect the likely future F-35 fuel usage. Further, one of the estimates did not use reasonable assumptions about part replacement rates and depot maintenance. Finally, while DOD took some steps to mitigate the uncertainties inherent in cost estimates, DOD officials did not conduct key analyses to determine the level of risk associated with the estimates.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD develop better informed affordability constraints; address three risks that could affect sustainment, affordability, and operational readiness; and take steps to improve the reliability of its cost estimates. DOD concurred with all but one recommendation and partially concurred with the recommendation to conduct uncertainty analysis on one of its cost estimates, stating it already conducts a form of uncertainty analysis. GAO continues to believe that the recommended analysis would provide a more comprehensive sense of the uncertainty in the estimates.

section_title: Letter
paragraphs: The Department of Defense (DOD) has a continuing responsibility to  procure weapon systems that successfully execute its national security  mission. One such weapon system is the F-35 Lightning II—also known  as the Joint Strike Fighter—which is intended to replace a variety of  existing aircraft in the Air Force, Navy, and Marine Corps, while providing  the most technologically advanced, lethal, supportable, and survivable  aircraft to date. With DOD estimating its sustainment costs to be  approximately $1 trillion over its life cycle, the F-35 is not only the most  ambitious weapon system in DOD’s history, but also the most costly.  Recognizing that at least 70 percent of a weapon system’s life-cycle costs  stem from operating and supporting the system, in recent years DOD has  made changes to its acquisition process to put an earlier emphasis on  sustainment. Specifically, more attention has been placed on sustainment  planning, assuring competition among suppliers, and identifying  sustainment-related resource constraints at the front end of the  acquisition process. However, the F-35 acquisition program, which began  in October 2001, predates these changes, and DOD has only recently  begun to focus on how it will sustain the nearly 2,500 aircraft it plans to  procure. Currently, the F-35 is 13 years into its acquisition strategy—a  strategy that involves substantial overlap among development, testing,  and production activities. With all three military services planning to  deploy and maintain the F-35 within the next 4 years, and the Marine  Corps planning to do so in less than 1 year, DOD is working to develop a  sustainment strategy for the F-35 that will be both affordable and  executable for the life cycle of the program.
We have reported on DOD’s acquisition of the F-35 for many years (see  the Related GAO Products section at the end of this report). Our body of  work has identified significant cost, schedule, and performance problems  and found that those problems, in large part, can be traced to (1)  decisions made at key junctures without adequate product knowledge;  and (2) a highly concurrent acquisition strategy with significant overlap  among development, testing, and manufacturing activities. In March  2012, DOD completed an extensive restructuring of the F-35 program by  increasing the program’s cost estimates, extending its testing and delivery  schedules, and deferring near-term aircraft procurement quantities into  the future. We concluded in June 2012 and in March 2013 that the  restructuring actions should lead to more achievable and predictable  outcomes, albeit at higher costs and with longer time frames than  originally planned for testing and delivering capabilities to the warfighter.  In March 2014, we found that problems encountered by DOD in  completing software testing may hinder delivery of expected warfighting  capabilities to most of the services. We have made numerous  recommendations aimed at addressing these issues, and DOD has taken  some actions to address them to varying degrees.
In light of DOD cost estimates for sustainment of the aircraft amounting to  approximately $1 trillion, and increased concerns over affordability in a  fiscally constrained environment, the House Armed Services Committee  report accompanying a National Defense Authorization bill for Fiscal Year  2014 mandated GAO to review DOD’s sustainment planning efforts for  the F-35 program.  This report addresses the extent to which DOD has  (1) developed a sustainment strategy for the F-35 program and  addressed potential risks to affordability and operational readiness and  (2) developed a reliable operating and support (O&S) cost estimate for  the life cycle of the program.
H.R. Rep. No. 113-102, at 118 (2013). and Evaluation, the Air Force, the Navy, the Marine Corps, and the F-35  Joint Program Office (JPO). We also gathered F-35 reliability and  maintainability (R+M) data from 2013 through 2014 that had been  verified through DOD’s Joint Reliability and Maintainability Evaluation  Team (JRMET). To determine the reliability of these data, we collected  information on how the data were collected, managed, and used through  a survey and interviews and with relevant DOD officials. In addition to the  data-reliability survey, we also reviewed the corresponding database user  manual and related documentation to determine the limitations of the  data. By assessing this information against GAO data-quality standards,  we determined that the data presented in our findings were sufficiently  reliable for presenting information about the aircraft’s reliability in this  report.
To interview officials about and observe F-35 operations, maintenance,  training, and developmental and operational testing, we conducted visits  to Eglin Air Force Base, Marine Corps Air Station Yuma, and Naval Air  Station Patuxent River—locations where these activities were occurring.  In addition, we met with officials about F-35 sustainment planning and  costs at Naval Air Systems Command and we reviewed documentation  and interviewed officials at Lockheed Martin—the prime contractor—in  Fort Worth, Texas, about the program’s status with respect to  development, operations, testing, and costs. To determine the extent to  which DOD has developed a sustainment strategy and addressed key  risks related to affordability and operational readiness, we reviewed  DOD’s guidance and policy for defense acquisitions and life-cycle  sustainment planning and compared them to the program’s Life Cycle  Sustainment Plan and other planning documents, including business- case analyses. Furthermore, we reviewed DOD’s risk-management and  policy guidance for specific sustainment elements and compared it to  current program risks and mitigation practices.
To determine the extent to which DOD has developed a reliable O&S cost  estimate for the F-35 program, we evaluated DOD’s two primary cost  estimates, the 2013 JPO and the 2013 CAPE office’s O&S cost  estimates, using GAO’s Cost Estimating and Assessment Guide. The  methodology outlined in the Cost Estimating and Assessment Guide is a  compilation of best practices that federal cost-estimating organizations  and industry use to develop and maintain reliable cost estimates  throughout the life of an acquisition program. The JPO’s estimate is used  to drive budget planning and is required by DOD acquisition policy, and  CAPE’s estimate is used to assess affordability and is required at major  acquisition milestones or by special request from DOD leadership.  Generally, either the JPO’s or CAPE’s estimate is used as DOD’s  estimate of record and published in the Selected Acquisition Report  (SAR)—a report that DOD is required to submit annually to Congress on  each of its major defense acquisition programs. For the F-35, DOD  decided to use the CAPE O&S estimate in the 2012 and 2013 SARs. We  evaluated both the JPO and the CAPE O&S estimates because both  estimates are used in DOD decision making. According to the Cost  Estimating and Assessment Guide, there are four general characteristics  of sound cost estimating: being well-documented, comprehensive,  accurate, and credible. For the purposes of this review, we conducted a  limited assessment and evaluated two of these characteristics:  comprehensive and credible. We chose to evaluate the estimates’  comprehensiveness because if a cost estimate is not comprehensive then  it cannot fully meet the characteristics of being well-documented or  accurate. For example, if the cost estimate is missing some cost  elements, then the documentation will be incomplete and the estimate will  be inaccurate. We also included an assessment of the credible  characteristic due to the substantial estimated cost of the program and to  determine whether risks to costs were quantified. To determine whether  the comprehensive and credibility characteristics were met, we reviewed  CAPE and JPO cost-estimating documentation, including data sources,  assumptions, and calculations, and we interviewed cost-estimating  officials from these offices.
We conducted this performance audit from August 2013 to September  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Appendix I provides a  more detailed description of our scope and methodology.
subsections: section_title: Background
paragraphs: The F-35 Lightning II program is a joint, multinational acquisition intended  to develop and field an affordable family of next-generation strike fighter  aircraft for the United States Air Force, Navy, Marine Corps, and eight  international partners. There will be three variants of the F-35. (1) The  conventional takeoff and landing (CTOL) variant, designated the F-35A,  will be a multirole, stealthy strike aircraft replacement for the Air Force’s  F-16 Falcon and the A-10 Thunderbolt II aircraft, and will complement the  F-22A Raptor (see fig. 1). (2) The short takeoff and vertical landing  (STOVL) variant, the F-35B, will be a multirole, stealthy strike fighter that  will replace the Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier  aircraft. (3) The carrier-suitable variant (CV), the F-35C, will provide the  Navy a multirole, stealthy strike aircraft to complement the F/A-18 E/F  Super Hornet. The Marine Corps will also field a limited number of F-35C  CVs. Lockheed Martin is the aircraft contractor and Pratt & Whitney is the  engine contractor.
DOD began the F-35 program in October 2001 with a highly concurrent  acquisition strategy, resulting in substantial overlap between  development, testing, and production. The program was restructured in  2003 and again in 2007 due to performance problems, cost growth, and  schedule slips. After the program experienced a critical cost breach in  2010, the program was restructured once more, and in 2012 some near- term procurement quantities were deferred to a future date and the  program’s cost estimates were increased. In 2013, DOD issued a report  to Congress stating that the Marine Corps, Air Force, and Navy were  planning to field initial operational capabilities (IOC) in 2015, 2016, and  2018, respectively. See figure 2 for a timeline of major events leading  from the beginning of development.
The program is currently in low-rate initial production, and the contractor  is assembling the next lot of 36 aircraft with a scheduled delivery by the  end of 2015. As of June 2014, 78 aircraft have been fielded, are flying,  and are being maintained at Eglin Air Force Base, Marine Corps Air  Station Yuma, Edwards Air Force Base, Nellis Air Force Base, and Luke  Air Force Base. The program had achieved about 18,250 cumulative flight  hours for the fleet of aircraft; a total of 200,000 flight hours are required in  order for the fleet to reach R+M maturity, as outlined in the program’s  Operational Requirements Document. In March 2014, we found that the  program had progressed in its developmental flight testing, but it  continued to lag in testing of critical mission-systems software, delaying  the delivery of expected warfighting capabilities.
By full-rate production, planned for fiscal year 2019, DOD would generally  be required to establish adequate sustainment and support systems for  the F-35.  Per DOD guidance for all weapon systems acquisitions, these  sustainment and support systems should be defined in a support concept  that is incorporated into a sustainment strategy. For the F-35, this concept  should comprise the necessary plans to conduct operations,  maintenance, and sustainment throughout the system’s life cycle, with the  F-35 Life Cycle Sustainment Plan serving as the principle document  governing F-35 sustainment. According to the F-35 Operational  Requirements Document, this concept must provide warfighting and  peacetime capability with the lowest cost of ownership, and all variants  must be able to deploy rapidly, sustain high mission reliability, and sustain  a high sortie-generation rate.
In developing this strategy, the program has made affordability its top  priority, and the F-35 Program Executive Officer reiterated this priority in  his April 2014 testimony. In this testimony, the Program Executive  Officer also identified areas in which the program is focusing on driving  down sustainment costs and identifying improvements to meet long-term  sustainment needs in order to “produce a mutually beneficial sustainment  enterprise that operates, manages, and supports the global system with  relevant metrics and incentives, while meeting warfighter-defined  readiness and cost objectives.” However, the department has stated that  it has a long history of starting programs that proved to be unaffordable,  and the result of this practice has been costly program cancellations and  dramatic reductions in inventory objectives. Recently updated DOD  acquisition policy has placed more emphasis on including sustainment  considerations as early as possible in the acquisition process.  According to the policy, a successful program meets the sustainment  performance requirements, remains affordable, and continues to seek  cost reductions throughout the Operations and Support Phase that begins  after the full-rate production decision.
In a 2012 memorandum, the Under Secretary of Defense for Acquisitions,  Technology and Logistics (AT&L) acknowledged that certain F-35  program areas including the Autonomic Logistics Information System  (ALIS), reliability growth, and sustainment costs require additional  improvement.
ALIS. ALIS is the primary sustainment tool for the F-35 and is  intended to predict maintenance and supply issues, automate logistics  support processes, and provide decision aids to help reduce life-cycle  sustainment costs and improve force readiness. ALIS is one of three  major components that make up the F-35 air system, along with the  aircraft and the engine, and comprises both hardware and software.  The F-35 program is delivering ALIS capabilities incrementally, which  include: operations, maintenance, supply chain, customer-support  services, training, tech data, system security and external interfaces.
Reliability and Maintainability (R+M). R+M measures aircraft  performance to determine how often the aircraft experiences failures  and how much time it takes to repair those failures. R+M is monitored  through a series of metrics that measure the intended performance of  the aircraft in meeting its requirements by maturity at a cumulative  200,000 flight hours with at least 50,000 flight hours per variant. R+M  drives sortie-generation rates and a reduced logistics footprint for the  F-35, as well as informing program O&S costs, which are tied to the  performance of the system at maturity.
O&S Costs. O&S consists of sustainment costs incurred from the  initial system deployment through the end of system operations and  includes all costs of operating, maintaining, and supporting a fielded  system. The F-35 JPO develops an annual estimate for the O&S  costs of maintaining and supporting the F-35 for 56 years.the estimate was $1.03 trillion; in 2012, it was $857 billion; and in  2013, it was $916 billion in then-year dollars. Additionally, CAPE  conducts independent cost estimates on certain programs at different  points in the acquisition cycle. In recent years, CAPE’s F-35 O&S cost  estimate has been reported in DOD’s annual SAR as the official O&S  cost estimate for the program. In 2013, that estimate was about $1.02  trillion in then-year dollars.
As stated earlier, recent changes to DOD’s acquisition guidance and  policy have also focused on making sustainment decisions earlier in the  acquisition process, by identifying the attributes of an effective  performance-based logistics arrangement and requiring strategies that  identify and manage technical data rights. Both of these areas will play an  important role in the future of the F-35 program.
Performance Based Logistics (PBL).based logistics strategy ties objective metrics for delivered logistical  system performance to incentives that will motivate the support  provider (i.e., the contractor). In recent years, F-35 program managers  have reaffirmed their commitment to employing a robust performance- based logistics system for the F-35 to help determine options for  pursuing the most value-driven performance-based agreements with  the support provider.
Technical Data Rights. Technical data for weapon systems include  the details necessary to ensure the adequacy of performance, as well  as instructions for operation, maintenance, and other actions needed  to support weapon systems. Technical data are an important part of  a weapon system program, such as the F-35. Identifying technical  data needs and costs early in the acquisition process can help  program managers maximize the potential for future competition and  develop the system’s overall sustainment strategy.
section_title: DOD Is Developing an F-35 Sustainment Strategy, but Key Risks to Affordability and Operational Readiness Remain
paragraphs: DOD’s development of an F-35 sustainment strategy is an ongoing  process that includes a number of plans, analyses, and decisions that  must be finalized by the start of full-rate production—estimated to occur in  fiscal year 2019. To date, the program’s sustainment strategy continues  to evolve but may not be affordable, and DOD has not fully addressed  several key risks to long-term affordability and operational readiness, that,  if not mitigated, could affect the development of the strategy.
subsections: section_title: DOD Is Currently Developing an F-35 Sustainment Strategy, but It May Not Be Affordable
paragraphs: DOD has a number of sustainment planning documents that will make up  the F-35 program’s overall sustainment strategy, including the Life Cycle  Sustainment Plan, which, according to DOD policy, will provide the basis  for sustainment activities. DOD also has past and ongoing studies and  analyses that have the potential to affect and inform the overall  sustainment strategy, including two Business Case Analyses and several  Level of Repair Analyses. These planning documents and analyses will  provide the basis for the overall sustainment strategy that will dictate  sustainment operations when the program enters full-rate production.  Furthermore, a recently launched initiative led by the F-35 JPO is working  to establish a roadmap to the end of the design phase in 2018, which is  intended to eventually result in a Future Support Construct—another  element of sustainment planning that will inform decisions for long-term  operations and support for the aircraft. The JPO plans to finalize the  Future Support Construct in March 2015.
However, the current sustainment strategy that DOD is developing may  not be affordable. The program has continued to experience cost  overruns, and the recent SAR estimated the O&S cost to sustain the  system for 56 years to be approximately $1 trillion. Additionally, according  to DOD officials, including officials within the Office of the Secretary of  Defense (OSD), the current sustainment strategy is not affordable.  According to CAPE analysis, the combined O&S costs of several legacy  aircraft—the F-15C/D, F-16C/D, AV-8B, and F-18A-D fleets—in 2010  exceeded $11 billion. Comparatively, based on CAPE’s 2013 O&S cost  estimate, the annual cost to sustain the F-35 will be about $19.9 billion (in  base year 2012 dollars) in 2040—the end of its steady-state years. This  $8.8 billion difference represents an increase of more than 79 percent in  annual O&S costs for the F-35 as compared with several legacy aircraft  (see fig. 3). Moreover, the Program Executive Officer has continued to  express concerns over the affordability of the program’s sustainment  approach, stating that “F-35 sustainment costs remain a concern” and  that affordability continues to be a top priority for the program.
In 2012, DOD established affordability targets for the program, stating  that the cost per flying hour for the Air Force, Marine Corps, and Navy F- 35 variants could not exceed $35,200, $38,400, and $36,300  respectively. However, DOD officials, including some at OSD, stated that  they believe that based on this threshold, the program is not affordable.  To address increasing costs, DOD has taken steps focused on cost  reduction including, but not limited to, establishing a Cost War Room in  2013 and a Readiness Cell in 2014. The Cost War Room is a  collaborative group comprising DOD, Lockheed Martin, and Pratt &  Whitney personnel, established with the purpose of reducing costs and  providing support for transitioning to performance-based logistics. The  mission of the Readiness Cell is to identify opportunities to enable F-35  availability (a sustainment metric) at more than 60 percent by 2015  across all three variants, thereby improving sustainment metrics that  directly affect O&S costs. However, it is unclear whether these cost- saving measures can result in an affordable approach to the long-term  operations and sustainment of the F-35 because the established  affordability targets that must be achieved may not be representative of  what the services can actually afford.
DOD’s current acquisition policy states that affordability constraints for  procurement and sustainment should be determined early in program  planning processes and should be used to ensure that capability  requirements are prioritized and that cost trade-offs occur as early as  possible and throughout the program’s life cycle. These constraints  should not be based on cost estimates but rather on whether a  component can afford the estimated costs of a system, and should be  used to indicate whether actions must be taken to prevent exceeding the  constraints. When DOD established affordability targets for the F-35  program in 2012, the methodology for determining that threshold was not  informed by actual resource constraints within service budgets at the  time. Specifically, DOD officials stated that the targets were determined  by arbitrarily lowering CAPE’s estimated F-35 Cost per Flight Hour by 10  percent. However, without informed affordability constraints, based on  military service budgets, DOD cannot be sure whether the costs savings it  achieves through current efforts will lead to an affordable sustainment  strategy, and DOD may miss additional areas for savings.
section_title: DOD Has Not Taken Steps to Fully Address Risks to Affordability and Operational Readiness That Could Affect the F-35 Sustainment Strategy
paragraphs: DOD faces the following three key risks to the long-term affordability and  operational readiness of the F-35 program: (1) the performance of ALIS;  (2) the R+M of the whole air system; and (3) the management of technical  data rights. However, the department has not taken steps to fully address  these risks, which could affect the development of the sustainment  strategy.
ALIS is a system of systems that serves as the primary logistics tool to  support operations, mission planning, and sustainment of the F-35. One  official described ALIS as the brains of the aircraft, helping maintainers  manage such things as aircraft health and diagnostics, supply-chain  management, and necessary maintenance events. However, ALIS has  experienced recurring problems, including user issues and schedule  delays. Furthermore, DOD has stated that ALIS is one of the biggest  impediments to the amount of sorties, or flights, that can be generated in  support of testing and operations. The sortie-generation rate is a key  performance parameter for the aircraft.
For example, the integration of ALIS capabilities—which are fielded in  increments—has been repeatedly delayed. The most recent timeline for  ALIS estimates that the final version of ALIS will be released for testing in  2017—7 years after the originally planned release date. Additionally,  ALIS’s diagnostic system has not reached full functionality. Specifically, it  has failed to meet basic requirements, including having the ability to  identify faults and failures in the aircraft. That issue has resulted in time- consuming workarounds that could further affect sortie generation.  Currently, DOD tracks issues with ALIS through an internal reporting  mechanism that allows users in the field to identify problems with the  system for potential fixes. For example, according to DOD officials, ALIS’s  Anomaly and Failure Resolution System (AFRS) is intended to identify  the necessary maintenance to repair the aircraft, but it is not yet mature.  Maintainers at an F-35 operational training site told us that, as a result,  they have had to use multiple approaches to identify the best  maintenance solution. Once identified, the maintainer can submit this  solution as an update for AFRS. However, this update must first be  reviewed by field support on site and then sent—in the form of an action  request—to the contractor for approval before it is integrated into AFRS.  Maintenance officials told us that they have submitted several thousand  action requests to date and have thereby created a backlog, leaving  maintainers to wait multiple days for an approval.
Problems with ALIS are reported by users on an individual basis, and  thus do not give DOD a complete view of ALIS’s system performance.  Once an issue is reported, DOD decides whether to (1) defer the issue,  forcing a workaround for users, or (2) pay for the contractor to develop a  “fix.” For an overview of this process, see figure 4.
To fully understand system performance, DOD’s Systems Engineering  Guide for Systems of Systems (SOS) states that it is important to have a  set of metrics that allow for an assessment of the SOS performance  traced to user requirements because the SOS will likely evolve based on  incremental changes in individual systems—similar to ALIS’s incremental  fielding. These metrics should measure the intended integrated  behavior and performance of the SOS in actual operations versus the  progress of the development of the SOS, allowing an assessment of SOS  capabilities based on user requirements, such as functionality between  interdependent subsystems. Additionally, according to the F-35  Operational Requirements Document, the F-35 must have a fully  functional and effective logistics system (e.g., ALIS) aimed at ensuring  operational readiness and availability and reducing overall life-cycle costs  by the time the aircraft reaches maturity at 200,000 flight hours for all  three variants with at least 50,000 per variant.
Department of Defense, Risk Management Guide for Acquisition (Washington, D.C.:  August 2006). performance parameters, such as sortie generation, to ensure operational  readiness and availability.
subsections: section_title: Reliability and Maintainability
paragraphs: DOD officials have continued to express concerns with the reliability and  maintainability (R+M) of the aircraft. In his April 2014 testimony, the  Program Executive Officer stated that R+M remains an area for needed  improvement. In that same month, the Director, Operational Test and  Evaluation (DOT&E), stated that the growth rates and improvement  programs that would be necessary to meet R+M requirements when the  aircraft reaches maturity (200,000 flight hours) are what they termed  “ambitious.” Additionally, DOT&E stated that future efforts to improve  R+M may be hampered by higher failure rates as the aircraft begins to fly  with more aggressive maneuvering and more extensive use of mission  systems.
To measure the R+M of the aircraft, testers collect data on (1) the number  of flight hours achieved before a failure occurs (reliability) and (2) the  amount of time it takes to repair those failures (maintainability). These two  primary measures are supported by various metrics and tracked along  planned growth curves to measure progress in meeting requirements.  Based on recent growth curves and reports for some of DOD’s R+M  metrics, some metrics have progressed, some continued to lag, and  some have worsened. For example:
Mean Flight Hours between Failures (Design Controllable) is the  average amount of flight hours achieved before a design-controllable  failure occurs. As of March 2014, this metric was progressing in that  the number of flight hours before a failure occurs was increasing for  all three variants. For example, the average flight hours between  failures for the F-35A—the variant with the most flight hours to date— was 5.2 in March 2014, surpassing the expectation at its current flight  hours by about 1.2 and growing toward its requirement at maturity of  6.0. Moreover, this was an increase of about 1.8 average flight hours  between failures since September 2013, as reported by GAO.
Mean Flight Hours between Critical Failures is the average amount  of flight hours achieved before a failure occurs that results in the loss   As of March  of a capability to perform a mission-essential function.2014, this metric was lagging well below its requirements at maturity,  meeting an average of 42 percent of those requirements across all  three variants.
Mean Time to Repair is the average time it takes a maintainer to  repair a failed component or device. Currently, this metric is not  improving in that as flight hours increase, it is taking maintainers  longer to repair failed components for the F-35A and F-35C, and the  amount of time it takes to repair failed components for the F-35B  remains unchanged. Specifically, GAO reviewed R+M growth curves  provided by DOD showing the historical growth of this metric from  2009 for the F-35B and 2010 for the F-35A and F-35C to March 2014,  and we observed that the metric is trending in the opposite direction of  its predicted path for the F-35A and F-35C, and the metric is  remaining steady, without improvement, for the F-35B.
A tenet of F-35 sustainment is to continually improve R+M of the weapon  system to drive down O&S costs. In an effort to improve R+M, DOD  began a Reliability Improvement Plan in 2014 that focused on addressing  the top 20 issues with the aircraft that affect R+M. While the current  Reliability Improvement Plan is in the process of addressing some R+M  issues, the plan focuses primarily on the aircraft’s hardware and does not  specifically address software and its potential effects on overall R+M. The  F-35 is the most software-intensive fighter aircraft DOD has procured to  date. According to DOD officials, software reliability is an issue that needs  attention; however, officials have also stated that the current process for  tracking R+M is almost exclusively dedicated to hardware.
To identify some software issues as they arise, users in the field use an  internal system to submit requests to the contractor, but these requests  are submitted on an individual basis and may not always be addressed  immediately as it takes time to determine whether the issue is related to  hardware or software. For example, officials told us that the Electrical  Optical Targeting System, which is used to track a target, continues to  fail. In this instance, testers reported the problem, and officials attempted  to improve the capability with hardware changes. However, not all issues  with the Electrical Optical Targeting System were fixed with the hardware  changes, and officials have decided to also try to address the issue with  software changes, causing users to identify workarounds in the  meantime. As another example, officials discussed instances in which the  diagnostics system signals to a maintainer that the landing gear failed,  but it was actually a sensor near the landing gear that failed. Because  software for isolating these types of failures is not yet mature, operators  and maintainers on the ground may continue to check the landing gear  without discovering the sensor issue and reporting it. Finally, officials  stated that with the release of the next increment of software in 2015, a  number of new issues may arise, and these issues may be related to  software because the new software includes new processors that can  affect mission systems on the aircraft.
According to DOD reliability guidance, basic reliability activities should  include software reliability assessments. Additionally, Naval Air Systems  Command guidance advises that weapon systems programs should  establish software R+M metrics as a key activity in terms of achieving  supportability goals and developing a high level of confidence that the  product will work reliably and can be maintained easily. Furthermore,  DOD’s Risk Management Guide states that effective risk management  depends, in part, on continuous monitoring and reassessment to identify  and address root causes of program risks. Currently there are no  processes or metrics by which the program can monitor and assess  software R+M and the full extent to which software issues may be  contributing to the overall R+M issues of the aircraft. Without an  assessment process, including metrics, for software R+M as a means of  mitigating ongoing risks to R+M, program officials cannot have a full  understanding of the root causes of R+M issues on the aircraft, or  whether software failures contribute to the overall R+M outside of  hardware failures. This can potentially lead to unforeseen costs in the  future as software is continually upgraded and refined. Since O&S costs  are based on meeting the required reliability at maturity (200,000 flight  hours), there are increasing risks to O&S cost and future aircraft  availability.
section_title: Technical Data
paragraphs: In May 2011, we found that DOD needs access to technical data— recorded information used to produce, support, maintain, or operate a  system—related to its weapon systems in order to help control costs  and maintain flexibility in the acquisition and sustainment of those  weapon systems. Technical data can enable the government to complete  maintenance work in-house, as well as to competitively award acquisition  and sustainment contracts. Additionally, we found that for service  contracts pertaining to DOD weapon programs, which can involve  products as well as support services, the lack of access to proprietary  technical data and a heavy reliance on specific contractors for expertise creates limitations to, or even precludes, the possibility of competition.
Within the past 5 years, acquisition reforms such as the Weapon Systems  Acquisition Reform Act of 2009 and other initiatives seeking greater  efficiency and cost savings in acquisitions have put greater emphasis on  obtaining technical-data rights and on maintaining competition throughout  the life cycle of weapon systems. In 2010, DOD announced its “Better  Buying Power” initiative that outlines a series of actions and directives to  promote competition, including ensuring that technical data requirements  are considered at key milestones of the acquisition cycle. Specifically, in  November 2010, the Under Secretary of Defense for Acquisition,  Technology and Logistics issued a memorandum that requires program  officials to (1) conduct a business case analysis that outlines the technical  data rights the government will pursue to ensure competition, and (2)  include the results of this analysis in certain strategies at Milestone B.
However, according to DOD officials, because the F-35 program’s initial  Milestone B decision occurred back in 2001, and these initiatives started  nearly a decade later, the F-35 program is getting a late start on its  assessment of its technical data needs.
In accordance with the Acquisition Strategy for the October 2001  Milestone B, the F-35 program did not acquire technical data suitable for  competition for the F-35 or its subsystems under its competitively  awarded system development contract. Furthermore, as stated in the  Acquisition Strategy, since the system design contract is in place, the  acquisition of data suitable for competition may be prohibitively expensive  and would not result in a commensurate return on investment. The  current iteration of the Acquisition Strategy leaves the door open for  acquiring current and future technical data that could support future  competition through the life cycle. Specifically, DOD plans to assess the  merits of acquiring operations, maintenance, installation, and training data  that would support sustainment and operation efforts. Planning for follow- on procurements is to include data rights/management strategies that will  support, to the extent practical, the options for competition through the  remainder of the program life cycle. With respect to this assessment, and  DOD’s current requirements related to technical data, DOD officials  confirmed the need for the F-35 program to begin identifying technical  data needs and costs related to future sustainment of the aircraft. They  said that DOD has begun discussions with contractor officials to  determine who, pursuant to the applicable contract clauses, has the rights  to specific technical data associated with the various hardware and  software systems. According to JPO officials, technical-data rights could  play a significant role in the development of the sustainment strategy for  the F-35; however, the federal government does not have an  understanding of the technical data it currently owns, what technical data  it may still need, and how much it may cost to acquire those data to  support the future sustainment of the F-35.
DOD, Defense Federal Acquisition Regulation Supplement (DFARS) Subpart 207.106  (S-70). The DFARS provides DOD implementation and supplementation of the Federal  Acquisition Regulation (FAR). The DFARS contains requirements of law, DOD-wide  policies, delegations of FAR authorities, deviations from FAR requirements, and policies  and procedures that have a significant effect beyond the internal operating procedures of  DOD or a significant cost or administrative effect on contractors. DFARS § 201.301. assess the long-term technical data and computer software needs  of those systems and subsystems; and establish acquisition strategies that provide for the technical data  and computer software deliverables and associated license rights  needed to sustain those systems and subsystems over their life  cycle; the strategy may include  the development of maintenance capabilities within DOD; or  competition for contracts for sustainment of the systems or  subsystems.
According to the current F-35 acquisition strategy, the F-35 program will  assess the merits of acquiring current and future technical data that could  support future competition throughout the life cycle, and will help provide  the basis for identifying the data and data rights required for the  sustainment of the program. However, as of July 2014, DOD officials had  not completed assessments of technical data rights, including a plan that  would identify what technical data rights the federal government currently  owns, what additional technical data rights it may need, and the costs  associated with purchasing any necessary technical data. According to  DOD officials, the program is planning to address these technical data  rights issues as part of the future sustainment strategy that will document  decisions for long-term operations and support for the aircraft. To date,  although DOD officials have acknowledged the importance of technical  data rights to the future of the F-35 program and have told us that  discussions with contractor officials regarding ownership of those rights  have begun, we have not seen any documentation confirming that  assessments of technical data rights will be completed. Until DOD  determines the technical data rights the federal government currently  owns and its critical technical data needs and associated costs, the  program runs the risk of being limited in its flexibility to make changes to  sustainment plans and the overarching sustainment strategy.  Furthermore, without a long-term IP Strategy, the program runs the risk of  not being able to compete requirements for such things as spare parts.  Unless technical data rights needs are considered up front, critical data  and software may not be acquired, rendering them unavailable (or  unaffordable) years later when seeking to maximize competition on a  program during its sustainment phase.
section_title: Improved Assumptions and Additional Analyses Could Increase Reliability of F-35 O&S Costs
paragraphs: DOD’s most recent O&S cost estimates are comprehensive in that they  include all DOD-required program elements and are organized according  to a standard O&S cost-estimating structure; but weaknesses exist with  respect to a few of the assumptions, and the estimates did not include all  analyses necessary to make them fully reliable.
subsections: section_title: DOD Created Two Primary Estimates of Operating and Support Costs for the F-35 across Its Life Cycle
paragraphs: There are two primary DOD F-35 O&S cost estimates, one developed by  the JPO and one by CAPE. O&S costs include the direct and indirect  costs of sustaining a fielded system, such as costs for replacement parts,  fuel, maintenance, personnel, support facilities, and equipment (see table  1). The JPO’s 2013 estimate, which is used to drive budget planning, lists  the total F-35 O&S costs across a 56-year life cycle as approximately  $916 billion in then-year dollars. CAPE’s estimate, which is used to  assess affordability and is required at major milestones or by special  request, lists the same cost to be about $1.02 trillion in then-year  dollars. Generally, either the JPO’s or CAPE’s estimate is used as  DOD’s estimate of record and published in the Selected Acquisition  Report (SAR)—a report that DOD is required to submit annually to  Congress on each of its major defense acquisition programs.
section_title: DOD F-35 Cost Estimates Are Comprehensive, but Weaknesses Exist in a Few Assumptions
paragraphs: The JPO and CAPE both created comprehensive F-35 O&S cost  estimates, but there are weaknesses in a few of the assumptions used in  the estimates. According to the GAO Cost Estimating and Assessment  Guide, a good cost estimate depends on how well the program is defined  in the technical baseline and work-breakdown structure. Both the JPO  and CAPE used a DOD-approved technical baseline and a product- oriented O&S cost work-breakdown structure to develop their 2013  estimates. The technical baseline provides the comprehensive program  description, which is used to determine what must be included in the  estimate. A work-breakdown structure is a necessary program- management tool because it provides a basic framework for estimating  costs, allowing for the estimates to be easily updated and compared to  future estimates. Both DOD guidance and the GAO Cost Estimating and  Assessment Guide highlight the importance of good documentation in  cost estimating, stating that a complete cost estimate should be formally  documented to serve as an audit trail of source data, methods, and  results. In addition, the GAO Cost Estimating and Assessment Guide  states that all assumptions and exclusions the estimate is based on  should be reasonable, clearly identified, and explained. While the JPO  and CAPE estimates are comprehensive, weaknesses exist because both  estimates used unreasonable assumptions for fuel burn rates and have  not clearly documented intermediate-level maintenance costs (see app. II  for more information on our analysis).
Fuel burn rates: The JPO and CAPE estimates do not use a  reasonable assumption for the Marine Corps fuel burn rate across the  life cycle of the aircraft. Both estimates use a fuel burn assumption of  1,493 gallons per flying hour for the Marine Corps variant. While the  rate used in the estimates was calculated based on service-planned  missions at maturity and DOD-validated physics-based models, this  burn rate is an almost exact match of the burn rate being observed at  the operational sites. This may be a reasonable assumption for the  short term, but it is likely that the fuel burn rates will increase in the  long term. The burn rate of 1,493 gallons per flying hour used in the  JPO and CAPE cost estimates reflects fuel burn data from aircraft  flown using limited capabilities—at slower speeds and at lower  altitudes than the F-35 will eventually fly. After 2015, the Marine Corps  is planning on using the aircraft’s increased capability, which will likely  result in more fuel being used. In addition, shortly after 2015, the  Marine Corps is planning on deploying the F-35 to ships, where the  aircraft’s fuel-intensive vertical landing capability will be used more  frequently, which will likely increase fuel usage. Consequently, the  use of lower fuel burn rate across the entire life cycle of the Marine  Corps variant is not a reasonable assumption.
Similarly, the JPO estimate lowered its fuel burn rate assumption to  1,480 gallons per flight hour from 1,558 gallons per flight hour for the  Air Force variant. However, the Air Force stated that the more  conservative assumption of 1,558 gallons per flight hour should be  used across the life cycle because the F-35 has yet to use its full flight  capabilities, weapons, or mission systems, which will likely increase  the fuel burn rates in the long term. The higher fuel burn rate  assumption for the Air Force would represent a $4.0 billion cost  increase in base year 2012 dollars across the life cycle of the aircraft.
Intermediate-level maintenance. These costs are not clearly  documented in either the JPO or the CAPE cost estimates. For the F- 35, intermediate-level maintenance tasks include wheel and tire  servicing, battery maintenance, structural airframe support, and  inspections. The documentation supporting DOD’s cost estimate  states that intermediate-level maintenance costs are not calculated for  the F-35, but the program’s maintenance planning documents state  that intermediate-level maintenance will be used in the sustainment of  the F-35, and it is currently being employed. Specifically, Marine  Corps officials told us that intermediate-level personnel at Yuma are  conducting some F-35 activities, including tire, wheel, and battery  servicing. In addition to the existing F-35 intermediate-level  maintenance being conducted at Yuma, DOD has conducted at least  five analyses to date, each resulting in a recommendation to repair a  F-35 Marine Corps variant component at the intermediate level. These  five component repairs represent more than $30 million in  intermediate-level maintenance costs across the life cycle of the  aircraft, but they are not documented in DOD’s cost estimates.  Moreover, all of the analyses to identify which components will be  repaired at the intermediate level are not yet complete, and the costs  related to intermediate-level maintenance could increase. According  to DOD officials, the costs of these intermediate-level repairs are  already included under mission personnel and unit-level maintenance  elements of the cost estimates. However, this is not documented in  the mission personnel or unit-level maintenance sections of the cost  estimates. While DOD guidance states that intermediate-level costs  can be combined with unit-level costs, the guidance also states that  this should be noted in the cost estimate’s supporting documentation  to avoid an interpretation that a portion of the maintenance costs were  omitted. Without clear documentation of these costs it is unclear  whether they are fully represented in the JPO and CAPE estimate.
In addition, the JPO estimate does not include reasonable assumptions  about part-replacement rates and depot maintenance.
Part replacement: The JPO estimate does not include reasonable  assumptions for part replacement. Based on data from the Air Force  and Marine Corps F-35 variants at testing and operational sites, parts  are being replaced, on average, 15 to 16 times more frequently than  the assumptions used across the life cycle of the JPO estimate (see  table 2). For example, a sensor that costs about $4,800 is being  replaced 60 to 129 times more frequently than anticipated across the  life cycle of the JPO cost estimate. Another example is the battery  charger unit, which costs about $60,000 to acquire new, and is being  replaced 3 to 8 times more frequently than anticipated across the life  cycle of the JPO cost estimate.
The part-replacement assumptions used by the JPO reflect the  anticipated reliability of the aircraft at maturity—once the entire fleet  has achieved 200,000 flight hours. According to JPO officials, the  reliability issues causing the high part-replacement rates will be  resolved once the aircraft reaches maturity, which is estimated to  occur at the end of fiscal year 2019. The JPO increased the cost of  replacing parts in the 2010 to 2019 portion of its estimate to reflect the  lower reliability of the aircraft until maturity. However, according to  officials from the Institute for Defense Analysis, who conducted a  study of the F-35’s R+M for DOT&E, the F-35 program would have to  achieve a higher reliability-growth improvement rate than has been  observed in almost all other aircraft in order to meet the anticipated  reliability by 2020. As previously stated, reliability improvement  efforts are under way that could reduce these costs, but it is unlikely  that these efforts will bring significant results in the near term because  the current F-35 fleet must be modified into the configuration  necessary for reliability improvement. As a result, for the next 10  years, the F-35 fleet will not represent the configurations necessary  for reliability-growth improvement. In addition, according to the study  of the F-35’s R+M conducted by the Institute for Defense Analyses for  DOT&E, future efforts to increase reliability may be hampered by  increased failure rates as the fielded fleet starts flying more  operationally representative missions, with more aggressive  maneuvering and more extensive use of mission systems.
CAPE adjusted its part-replacement assumptions for its 2013 estimate  to reflect higher replacement rates across the life cycle of the aircraft.
As a result, CAPE’s 2013 estimated cost for parts across the life cycle  of the program is $23 billion greater than the JPO’s in base year 2012  dollars. CAPE officials told us that if they had used the actual  replacement rates being observed at the F-35 sites, the part- replacement cost across the life cycle of the program would have  been roughly $120 billion greater than the JPO’s in base year 2012  dollars.
Depot maintenance: Finally, the JPO estimate does not include a  reasonable assumption regarding F-35 depot maintenance. The depot  maintenance cost element includes the labor, materiel, and overhead  costs for overhaul or rework of aircraft returned to a government or  contractor repair facility. This O&S cost includes scheduled and  unscheduled depot maintenance, as well as inspections. We have  found this to be a cost element that has been underestimated in the  past for legacy and fifth-generation aircraft, including the F/A-18 and  F-22. The JPO reduced its assumptions related to the cost of depot  maintenance by about half, from about $2 million per scheduled depot  induction in the 2012 estimate to about $600,000–$1.3 million in the  2013 estimate. This change resulted in a $19.7 billion reduction in  then-year dollars from the JPO’s 2012 estimate. JPO officials stated  that this reduction was the result of using a different cost-estimation  method, which had the effect of lowering the hourly cost per induction.  According to the assumptions in the JPO’s 2013 estimate, the  average cost per depot induction hour is roughly $205. However, in  2004, the average cost per depot induction was about $238, and it is  likely that labor and materiel costs have increased over the past 10  years. To develop its hourly cost, the JPO used F-16 data from  Lockheed Martin contractors that had been adjusted for the F-35.  According to the JPO’s current assumptions, materiel costs would be  30 percent of the labor costs, but data for the AV-8 and F/A-18 depot  inductions used in the previous year’s estimate indicate that depot  induction materiel costs have historically been closer to 45 percent of  labor costs. As a result, it is likely that the depot maintenance hourly  cost used in the JPO’s 2013 estimate is not a reasonable assumption.
section_title: DOD Has Not Conducted Uncertainty Analyses to Determine Credibility of Estimates
paragraphs: Every cost estimate contains a degree of uncertainty because of the  many assumptions that must be made about the future. To mitigate this  uncertainty, a variety of checks and analyses can be conducted to  determine the credibility of the assumptions and the estimate as a whole.  While the JPO and CAPE have done some cross-checks and sensitivity  analyses to mitigate the uncertainty of their assumptions, they have not  conducted uncertainty analyses to determine whether the estimates  reflect the most likely costs.
According to the GAO Cost Estimate and Assessment Guide and DOD  guidance on cost estimating, a good practice for determining the  reasonableness of an estimate’s assumptions for certain cost elements is  to cross-check them with other programs’ estimates or other cost- estimate methodologies to determine whether results are similar. JPO  and CAPE cost estimators stated that they cross-checked their results of  key cost elements against O&S estimates for other aircraft to determine  whether they produced similar results. For example, according to a JPO  cost-estimating official, the F-35 results for the depot maintenance cost  element were three to five times greater than the same costs for legacy  aircraft. In addition, CAPE conducted a cross-check comparing the cost  per flying hour of legacy and F-35 aircraft, which identified F-35 costs as  being significantly higher than the legacy aircraft. The official stated that  this was reasonable given the complexity of the fifth-generation aircraft  and the additional capability that the F-35 offers.
The JPO and CAPE have also conducted some sensitivity analyses, but  they have not conducted uncertainty analyses to determine the credibility  of their estimates. Sensitivity analyses examine how changes to individual  assumptions and inputs affect the estimate as a whole. Both the JPO and  CAPE cost estimators have told us that they conducted “what if”  scenarios—varying the assumptions of certain variables individually to  see how the total O&S costs may be affected. Although CAPE did not  conduct a sensitivity analysis on its 2013 O&S cost estimate, it did  conduct one for its 2011 F-35 O&S cost estimate, which contains many of  the same assumptions as its 2013 estimate. However, as a best practice,  a sensitivity analysis should be included in all cost estimates because it  examines the effects of changing assumptions and ground rules. The  JPO asked officials at the Office of the Secretary of Defense, Logistics  and Materiel Readiness (OSD-LMR), to conduct a sensitivity analysis  using the JPO cost model. OSD-LMR conducted this analysis,  determining the cost elements that represented the highest percentage of  cost and varying assumptions to determine which factors have the  greatest effect on the total O&S cost. According to DOD guidance,  sensitivity analyses are useful for identifying critical estimating  assumptions, but they have limited utility in providing a comprehensive  sense of overall uncertainty. In contrast, an uncertainty or quantitative risk  analysis can provide a broad overall assessment of variability in the cost  estimate. Similarly, the GAO Cost Estimating and Assessment Guide  states that risk and uncertainty analyses should be performed to  determine the level of risk associated with the estimate. JPO and CAPE  cost estimators have not conducted uncertainty analyses to understand  the potential range of costs around their point estimates and to convey  their level of confidence that the estimates reflect the most likely costs.  JPO and CAPE officials stated that they did not conduct uncertainty  analyses because they are not a standard practice for DOD when  creating O&S cost estimates. However, according to the GAO Cost  Estimating and Assessment Guide, an estimate without a risk and  uncertainty analysis is unrealistic because it does not assess the  variability in the estimate.
In April 2014 testimony, the F-35 program executive officer highlighted  the importance of refining the cost estimate as a key component of DOD’s  risk-reduction efforts related to the F-35. Without fully reliable O&S cost  information to develop accurate budget requests and address the  affordability of the F-35 program, the military services may be at risk of  sustainment cost overruns in the near and long terms. The use of  improved assumptions, as well as an understanding of the level of  confidence in the estimates through the use of an uncertainty analysis,  would also improve DOD’s ability to make F-35 sustainment decisions.
section_title: Conclusions
paragraphs: The F-35 program remains one of the top priorities for the future of our  national defense, according to senior defense leadership. However, as  we have previously reported, the F-35 program has experienced cost,  schedule, and performance problems that have put it 7 years behind  schedule and significantly over its original budget. With life-cycle  sustainment costs estimated by DOD at about $1 trillion, the F-35 is  approaching a critical juncture. In an era of significant budgetary  pressures and competition for resources, with the Air Force, Navy, and  Marine Corps each beginning to deploy and maintain F-35 aircraft within  the next 4 years, and with full-rate production planned in fiscal year 2019,  DOD must find ways to make the sustainment of the F-35 program  affordable. DOD has begun to devote attention to sustainment through  the creation of plans and analyses that could help inform programmatic  decisions and shape its overall sustainment strategy. One key decision is  for DOD to determine what it can realistically afford with respect to the F- 35 program, but until it identifies affordability constraints tied to the  military services’ budgets, it will continue to develop and field the most  costly weapon system program in history without knowing whether the Air  Force, Navy, or Marine Corps can pay for it. Furthermore, while DOD has  begun testing and fielding the F-35 at sites around the country, the  program faces several risks, including ALIS, software R+M, and  uncertainties related to technical data rights, which could adversely affect  DOD’s sustainment strategy. If these risks remain unaddressed,  sustainment costs could potentially increase well beyond current  estimated levels, and operational readiness could suffer.
Finally, although DOD’s current cost estimates identify most sustainment  costs and document cost-influencing assumptions, certain elements may  be underestimated. In some areas, such as the assumptions underlying  future fuel burn rates, continuing to underestimate these elements could  translate to billions of dollars in cost increases over the life cycle of the  program. Therefore, it is imperative that DOD provide an accurate picture  of the current and future state of the program, as well as a greater degree  of assurance that its cost estimates are credible—through the use of  uncertainty analyses—to help inform the sustainment strategy. With more  than 2,400 F-35 aircraft still planned for purchase, DOD has an  opportunity to chart a more informed, analytical, and affordable path for  the program. However, sustainment traditionally represents about 70  percent of a weapon system’s life-cycle cost, and until DOD develops a  viable sustainment strategy, addresses key risks, and improves its cost  estimates, the department will likely continue to present Congress with  near-term funding requests without having full knowledge of the extent of  the program’s long-term financial requirements.
section_title: Recommendations for Executive Action
paragraphs: We are making the following eight recommendations to improve DOD’s  sustainment planning and operating and support (O&S) cost estimates for  the F-35 program.
To help DOD develop an affordable sustainment strategy for the F-35, we  recommend that the Secretary of Defense direct the Under Secretary of  Defense for Acquisitions, Technology and Logistics to direct the F-35  Program Executive Officer to establish affordability constraints linked to,  and informed by, military service budgets that will help guide sustainment  decisions, prioritize requirements, and identify additional areas for  savings by March 2015, at which point the Future Support Construct  decision will be approved.
To help DOD address key risks to F-35 affordability and operational  readiness, and to improve the reliability of its O&S cost estimates for the  life cycle of the program, we recommend that the Secretary of Defense  direct the F-35 Program Executive Officer to take the following five  actions:
To enable DOD to better identify, address, and mitigate performance  issues with the Autonomic Logistics Information System (ALIS) that  could have an effect on affordability, as well as readiness, establish a  performance-measurement process for ALIS that includes, but is not  limited to, performance metrics and targets that (1) are based on  intended behavior of the system in actual operations and (2) tie  system performance to user requirements.
To develop a high level of confidence that the aircraft will achieve its  R+M goals, develop a software reliability and maintainability (R+M)  assessment process, with metrics, by which the program can monitor  and determine the effect that software issues may have on overall F- 35 R+M issues.
To promote competition, address affordability, and inform its  overarching sustainment strategy, develop a long-term Intellectual  Property (IP) Strategy to include, but not be limited to, the  identification of  current levels of technical data rights ownership by the federal  government and all critical technical data needs and their associated costs.
To improve the reliability of the JPO F-35 O&S cost estimate, clearly  document assumptions related to intermediate-level maintenance  costs and revise assumptions related to fuel burn rates, part  replacement, and depot-maintenance induction in its future F-35 O&S  cost estimates to better reflect the current and future state of the F-35  program.
To understand the potential range of costs associated with the JPO F- 35 O&S cost estimate, conduct uncertainty analyses on future JPO  estimates.
To improve the reliability of the Cost Assessment and Program Evaluation  (CAPE) F-35 O&S cost estimate, we recommend that the Secretary of  Defense direct the Director of CAPE to take the following two actions for  future F-35 O&S cost estimates:  clearly document assumptions related to intermediate-level  maintenance and revise fuel burn assumptions to better reflect the  current and future state of the F-35 program and  conduct uncertainty analyses to understand the potential range of  costs associated with its estimates to reflect the most likely costs  associated with the program.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with seven  of the report’s eight recommendations and partially concurred with one  recommendation.  DOD’s comments are summarized below and reprinted  in appendix III.  DOD also provided technical comments, which we have  incorporated into our report where appropriate.
DOD concurred with the recommendation that the Secretary of Defense  direct the Under Secretary of Defense for Acquisitions, Technology, and  Logistics to direct the F-35 Program Executive Officer to establish  affordability constraints linked to, and informed by, military service  budgets.  DOD stated that it established F-35 affordability targets for unit  recurring flyaway cost and for sustainment cost in March 2012 to guide  development of an affordable strategy.  DOD further stated that the F-35  program is currently operating under these affordability targets and using  affordability goals to reduce life-cycle costs.  We acknowledge that the  department established affordability targets for sustainment in March  2012, but these actions do not fully address the intent of our  recommendation because, as we note in our report, these affordability  targets may not be representative of what the services can actually afford  because the methodology for determining the targets was not informed by  resource constraints within military service budgets.  We further  acknowledge in our report that DOD is currently engaged in cost- reduction efforts, but until DOD establishes affordability targets that are  informed by military service budgets, the department cannot be sure  whether the cost savings it achieves through current efforts will lead to an  affordable sustainment strategy.  If DOD develops affordability targets  that are linked to and driven by military service budgets, this action should  address the recommendation.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to establish a performance- measurement process for ALIS that includes, but is not limited to,  performance metrics and targets that (1) are based on intended behavior  of the system in actual operations and (2) tie system performance to user  requirements.  DOD stated that the F-35 program will continue to use  ALIS performance metrics and targets to assess effects on the  operational requirements, such as the sortie-generation rate key  performance parameter, through the remainder of system development  and demonstration (SDD).  DOD also stated that while the F-35 program  continues to assess these effects today, future incremental ALIS  functional deliveries and additional operational experience will enable  more-refined metrics over time which will help reduce performance risk  and ensure a successful operational test and evaluation at the end of  SDD.  As our report states, we acknowledge that metrics for ALIS’s  availability are used to track the development of ALIS, but these metrics  do not fully address the intent of our recommendation because there is no  performance-measurement process that directly ties ALIS performance to  user requirements.  While the sortie-generation rate key performance  parameter is one way in which to monitor performance, this metric is  aimed at measuring the performance of the entire F-35 air system and  not, specifically, how ALIS performs its functions in response to user  requirements.  We agree that, as DOD noted, performance metrics and  targets may be refined over time to help reduce performance risk, but  until DOD develops performance targets and metrics tied directly to ALIS  functionality and user requirements, it may still face additional costs and  risks to operational readiness and availability.  If DOD develops these  performance targets and metrics as part of a performance-measurement  process, this action should address our recommendation.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to develop a software reliability  and maintainability assessment process, with metrics, by which the  program can monitor and determine the effect that software issues may  have on overall F-35 reliability and maintainability (R+M) issues. DOD  stated that the F-35 program has an established process for assessing  software reliability and will continue to monitor and determine the effect  that software issues may have on overall F-35 R+M. DOD noted that this  process includes a Failure Reporting Analysis Corrective Action System  database, a Joint Reliability and Maintainability Evaluation Team, and a  Test Data Scoring Board, and uses the Mean-Flight-Hours-Between- Failure metric.  Although we agree that the components DOD lists are in  place, and that DOD does have a process for assessing F-35 R+M, these  actions do not fully address the intent of our recommendation because,  as we state in our report, this process is aimed primarily at assessing the  reliability of the F-35 aircraft’s hardware and not, specifically, its  associated software.  Specifically, the process does not determine and  monitor software reliability using software-specific metrics and targets.   We continue to believe that DOD’s current process and associated  metrics do not allow program officials to have a full understanding of the  root causes of reliability issues and the extent to which software failures  contribute to these issues.  As our report states, the reliability of the  aircraft is measured with a number of different metrics; however,  according to DOD officials, the current process and associated metrics  are almost exclusively dedicated to hardware.  Since the F-35 is the most  software-intensive aircraft DOD has procured to date, establishing a  software reliability assessment process with metrics and targets, outside  of hardware reliability, would allow DOD to determine the extent to which  software failures contribute to the overall R+M of the F-35.  If DOD  develops such a process, this action should address our  recommendation.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to, in order to promote  competition, address affordability and, to inform its overarching  sustainment strategy, develop a long-term IP Strategy to include, but not  be limited to, the identification of (1) current levels of technical data rights  ownership by the federal government and (2) all critical technical data  needs and their associated costs.  DOD stated that the F-35 acquisition  strategy includes a data-management strategy whereby it will continue to  assess the acquisition of technical data as specific needs arise  throughout the life cycle and will determine the level of specified rights,  critical technical data needs and associated costs, through business-case  analysis, contract strategy development, and ongoing sustainment  planning.  If DOD develops a long-term IP strategy that includes the  identification of its current level of specified rights, critical technical data  needs, and associated costs, this action should address the  recommendation.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to clearly document  assumptions related to intermediate-level maintenance costs and revise  assumptions related to fuel burn rates, part replacement, and depot- maintenance induction in its future F-35 O&S cost estimates. DOD stated  that the Program Executive Officer will document intermediate-level  maintenance cost assumptions and revise fuel burn rates, part  replacement, and depot-maintenance induction assumptions in future  annual updates to the F-35 O&S cost estimate.  We agree that, if the  assumptions are revised and fully documented to reflect the current and  future state of the program, this action should address the  recommendation.
DOD concurred with the recommendation that the Program Executive  Officer conduct uncertainty analysis on future F-35 O&S cost estimates.  DOD stated that it will ensure that the Program Executive Officer  conducts risk and sensitivity analysis, in accordance with DOD guidance,  in future annual updates to the cost estimate. If the Program Executive  Officer performs an uncertainty or quantitative risk analysis on future JPO  F-35 O&S estimates, this action should address the recommendation.
DOD concurred with the recommendation that CAPE clearly document  assumptions related to intermediate-level maintenance and revise its  O&S cost estimate’s fuel burn assumptions to better reflect the current  and future state of the F-35 program. DOD stated that CAPE’s current  documentation of the O&S cost estimate reflects an approved two-level  maintenance strategy (i.e., organizational- and depot-level maintenance  only, no intermediate-level maintenance). DOD also stated that CAPE will  ensure that if the strategy is revised to include any planned intermediate- level maintenance activities, the activities and associated costs will be  fully captured within program estimates and documentation. However, the  program’s maintenance planning documents already state that  intermediate-level maintenance will be used in the sustainment of the F- 35, and it is currently being employed. In its written response, DOD  further stated that any intermediate-level costs are already included in the  estimate’s mission personnel costs. While DOD guidance states that  intermediate-level costs can be combined with unit-level costs, including  mission personnel, the guidance also states that this should be noted in  the cost estimate’s supporting documentation to avoid an interpretation  that a portion of the maintenance costs were omitted.  However, as we  note in the report, CAPE did not document intermediate-level  maintenance costs in the mission personnel or unit-level maintenance  sections of the cost estimates. Without clear documentation of these  intermediate-level maintenance costs, it is unclear whether they are fully  represented in the CAPE estimate. If CAPE clearly documents where and  how intermediate maintenance costs are included in its future cost  estimates, this action should address the recommendation.
Related to revising its O&S cost estimate’s fuel burn assumptions, DOD  stated that CAPE will update its estimate to reflect the latest approved  technical baseline and incorporate the latest approved fuel consumption  actuals, as this information becomes available. As our report states, the  estimate’s fuel burn rate assumption should reflect the most likely future  state of the program across its life cycle. To date, fuel consumption  actuals are from aircraft flown using limited capabilities—at slower speeds  and at lower altitudes than the F-35 will eventually fly—and do not reflect  the most likely future state of the program.  We agree that, once fuel  consumption data from aircraft flown using the full capability are available,  these data could be used to update the assumptions. Until then, the  estimate’s fuel burn rate assumptions should be revised to reflect the  future state of the program. If fully implemented, DOD’s proposed actions  to update its estimate should address the recommendation.
Finally, DOD partially concurred with the recommendation that CAPE  conduct uncertainty analyses to understand the total potential range of  costs and to determine if its estimate reflects the most likely costs  associated with the program. DOD stated that the department agrees with  the value of understanding potential cost ranges, but stated that it  considers the risk and sensitivity analyses regularly performed by CAPE  to be a form of uncertainty analysis.  As our report states, we  acknowledge that CAPE has conducted some sensitivity analyses on  previous F-35 O&S estimates, but it has not conducted an uncertainty or  quantitative risk analysis on its F-35 O&S cost estimate.  According to  DOD guidance, sensitivity analyses are useful for identifying critical  estimating assumptions, but they have limited utility in providing a  comprehensive sense of overall uncertainty. In contrast, an uncertainty or  quantitative risk analysis, such as a Monte Carlo simulation, can provide  a broad overall assessment of variability in the cost estimate. Similarly,  the GAO Cost Estimating and Assessment Guide states that risk and  uncertainty analyses should be performed to determine the level of risk  associated with an estimate. We continue to believe that without an  uncertainty analysis, DOD cannot fully understand how much of the total  potential O&S costs are accounted for in its nearly $1 trillion estimate. If  CAPE performs an uncertainty analysis on future estimates, this action  should address our recommendation.
We are sending copies of this report to appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Air Force,  Army, and Navy; and the Commandant of the Marine Corps. In addition,  the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff has any questions about this report, please contact me  at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Staff members making key contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To address both of our objectives—that is, to evaluate F-35 sustainment  planning and risk and operating and support (O&S) cost estimates—we  collected and analyzed information and interviewed officials from the  following Department of Defense (DOD) offices:
Office of the Under Secretary of Defense (Acquisitions, Technology
Office of the Assistant Secretary of Defense (Logistics and Materiel
Office of the Director for Cost Assessment and Program Evaluation
Office of the Director for Operational Test and Evaluation (DOT&E);
Office of the Deputy Assistant Secretary of Defense for  Developmental Test and Evaluation;
Office of the Deputy Assistant Secretary of Defense for Systems
Department of the Air Force;
Department of the Navy;
Headquarters Marine Corps; and
F-35 Joint Program Office (JPO).
Eglin Air Force Base, Florida;
Marine Corps Air Station Yuma, Arizona;
Naval Air Station Patuxent River, Maryland;
Naval Air Systems Command, Maryland; and
Lockheed Martin, Fort Worth, Texas.
To assess the extent to which DOD has developed a sustainment  strategy for the F-35 program and addressed potential risks related to  affordability and operational readiness, we reviewed documentation of  program plans and analyses with relevant sustainment elements,  including the F-35 Life Cycle Sustainment Plan, the Weapon System  Planning Document, the F-35 Future Support Construct, the F-35  Autonomic Logistics Global Sustainment Concept of Operations, and the  F-35 Operational Requirements Document. We reviewed documentation  of ongoing analyses, including Level of Repair Analyses and Business  Case Analyses. We compared these documents to DOD acquisition  guidance and policy on sustainment planning, including the Defense  Acquisition Guidebook, Interim DOD Instruction 5000.02 “Operation of the  Defense Acquisition System,” and DOD’s Life Cycle Sustainment Plan  and guidance.
To determine the extent to which DOD has developed plans to mitigate  sustainment risks, we conducted site visits (see locations above) and  interviewed key DOD officials in order to collect information about  ongoing operations, testing, and risks to the sustainment of the F-35.  After identifying the three most significant risks to the sustainment of the  program, we compared DOD’s risk-mitigation efforts to DOD Risk  Management Guide and other DOD acquisition guidance and DOD policy  and guidance. Specifically, we reviewed DOD’s efforts to monitor and  improve the risks of (1) the performance of the Autonomic Logistics  Information System (ALIS), (2) the reliability and maintainability (R+M) of  the aircraft, and (3) the management of technical data. We then  compared these efforts to DOD’s Risk Management Guide, Interim DOD  Instruction 5000.02, and other specific guidance and best practices,  including DOD’s System of Systems Guide, DOD’s Manual for Improving  Reliability, Availability, and Maintainability, and Naval Air Systems  Command Software Logistics Primer.
To determine the extent to which DOD has developed a reliable operating  and support (O&S) cost estimate for the F-35 program, we evaluated both  DOD’s Joint Program Office (JPO) and its Cost Assessment and Program  Evaluation (CAPE) office 2013 O&S cost estimates using GAO’s Cost  Estimating and Assessment Guide. Specifically, the methodology  outlined in the Cost Estimating and Assessment Guide is a compilation of  best practices that federal cost-estimating organizations and industry use  to develop and maintain reliable cost estimates throughout the life cycle  of an acquisition program. The JPO’s estimate is used to drive budget  planning and CAPE’s estimate is used to assess affordability and is  required at major milestones or by special request. Generally, either the  JPO’s or CAPE’s estimate is used as DOD’s estimate of record and  published in the Selected Acquisition Report (SAR)—a report that DOD is  generally required to submit annually to Congress on each of its major  defense acquisition programs. For the F-35, DOD decided to use the  CAPE O&S estimate in the 2012 and 2013 SARs. We evaluated both the  JPO and the CAPE O&S estimates because both estimates are used in  DOD decision making. According to the GAO Cost Estimating and  Assessment Guide, there are four general characteristics of sound cost  estimating: being well-documented, comprehensive, accurate, and  credible. For the purposes of this engagement, we conducted a limited  assessment and evaluated two of these characteristics: comprehensive  and credible. We chose to evaluate the estimates’ comprehensiveness  because if a cost estimate is not comprehensive then it cannot fully meet  the characteristics of being well-documented or accurate. For example, if  the cost estimate is missing some cost elements, then the documentation  will be incomplete and the estimate will be inaccurate. We also included  an assessment of the credible characteristic due to the substantial  estimated cost of the program and the need to better understand and  quantify the F-35 O&S cost risk. To determine whether the  comprehensive and credibility characteristics were met, we reviewed  CAPE and JPO cost-estimating documentation, including data sources,  assumptions, and calculations, and we interviewed cost-estimating  officials from these offices. In addition, to determine how much more  frequently parts were being replaced than assumed in the JPO cost  estimate (see table 2), we compared the mean flight hour between  removals (MFHBR) data for the Air Force’s F-35A (CTOL) and the Marine  Corps’ F-35B (STOVL) from a 12-month period ending in March 31, 2014,  to the MFHBR assumptions used in the JPO estimate.
We also gathered F-35 reliability and maintainability (R+M) data2013 through 2014 that had been verified through DOD’s Joint Reliability  and Maintainability Evaluation Team (JRMET). To determine the reliability  of these data, we collected information on how the data were collected,  managed, and used through a survey and interviews with relevant DOD  officials. In addition to the data-reliability survey, we also reviewed user  manual and related documentation to determine the limitations of the  data. By assessing this information against GAO data-quality standards,  we determined that the data presented in our findings were sufficiently   from  reliable for presenting information about the aircraft’s reliability in this  report.
We conducted this performance audit from August 2013 through  September 2014 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: GAO Scoring of 2013 F-35 Operating and Support (O&S) Estimates
paragraphs: We completed an overall assessment of the Joint Program Office (JPO)  and Cost Assessment and Program Evaluation (CAPE) F-35 O&S  estimates on the basis of two characteristics—comprehensiveness and  credibility—and their associated best practices derived from the GAO  Cost Estimating and Assessment Guide. After reviewing documentation  that JPO and CAPE submitted for their 2013 F-35 O&S cost estimates,  conducting interviews with JPO and CAPE cost-estimating officials, and  reviewing relevant sources, we determined that these cost estimates are  not fully reliable. While the estimates are comprehensive, not all of the  assumptions that the estimates were based on were clearly documented  or reasonable. In addition, the estimates are not fully reliable in that the  JPO estimate is partially credible and the CAPE estimate is minimally  credible. These evaluations are shown in table 3 below. We determined  the overall assessment rating by assigning each individual best practice  rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3,  Substantially Met = 4, and Met = 5. Then, we took the average of the  individual best practice assessment ratings to determine the overall rating  for each of the two characteristics. The resulting average becomes the  Overall Assessment as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5  to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met =  4.5 to 5.0. A cost estimate is considered reliable if the overall assessment  ratings for each of the two characteristics are substantially or fully met. If  any of the characteristics are not met, minimally met, or partially met, then  the cost estimate does not fully reflect the characteristics of a high-quality  estimate and cannot be considered reliable.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff members  made key contributions to this report: Alissa Czyz, Assistant Director;  Jennifer Echard; Jeffrey Hubbard; Abishek Krupanand; Jason Lee; Jennie  Leotta; Katie Mauldin; Carol Petersen; Amie Steele; Alyssa Weir; Cheryl  Weissman; and Amanda Weldon.
section_title: Related GAO Products
paragraphs: Weapon Systems Management: DOD Has Taken Steps to Implement  Product Support Managers but Needs to Evaluate Their Effects.  GAO-14-326. Washington, D.C.: April 29, 2014.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-14-340SP. Washington, D.C.: March 31, 2014.
F-35 Joint Strike Fighter: Slower Than Expected Progress in Software  Testing May Limit Initial Warfighting Capabilities. GAO-14-468T.  Washington, D.C.: March 26, 2014.
F-35 Joint Strike Fighter: Problems Completing Software Testing May  Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322.  Washington, D.C.: March 24, 2014.
F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but  Affordability Challenges and Other Risks Remain. GAO-13-690T.  Washington, D.C.: June 19, 2013.
F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but  Affordability Challenges and Other Risks Remain. GAO-13-500T.  Washington, D.C.: April 17, 2013.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-13-294SP. Washington, D.C.: March 28, 2013.
F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term  Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March  11, 2013.
Fighter Aircraft: Better Cost Estimates Needed for Extending the Service  Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.:  November 15, 2012.
Joint Strike Fighter: DOD Actions Needed to Further Enhance  Restructuring and Address Affordability Risks. GAO-12-437. Washington,  D.C.: June 14, 2012.
Tactical Aircraft: F-22A Modernization Program Faces Cost, Technical,  and Sustainment Risks. GAO-12-447.Washington, D.C.: May 2, 2012.
Joint Strike Fighter: Restructuring Added Resources and Reduced Risk,  but Concurrency Is Still a Major Concern. GAO-12-525T. Washington,  D.C.: March 20, 2012.
Defense Logistics: Improvements Needed to Enhance Oversight of  Estimated Long-term Costs for Operating and Supporting Major Weapon  Systems. GAO-12-340. Washington, D.C.: February 2, 2012.
Joint Strike Fighter: Implications of Program Restructuring and Other  Recent Developments on Key Aspects of DOD’s Prior Alternate Engine  Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing,  but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.:  May 19, 2011.
Defense Acquisition: DOD Should Clarify Requirements for Assessing  and Documenting Technical-Data Needs. GAO-11-469. Washington,  D.C.: May 11, 2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.
Joint Strike Fighter: Restructuring Should Improve Outcomes, but  Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.:  March 15, 2011.
Defense Management: DOD Needs to Monitor and Assess Corrective  Actions Resulting from Its Corrosion Study of the F-35 Joint Strike  Fighter. GAO-11-171R. Washington, D.C.: December 16, 2010.
Tactical Aircraft: DOD’s Ability to Meet Future Requirements Is Uncertain,  with Key Analyses Needed to Inform Upcoming Investment Decisions.  GAO-10-789. Washington, D.C.: July 29, 2010.
Joint Strike Fighter: Significant Challenges and Decisions Ahead.  GAO-10-478T.Washington, D.C.: March 24, 2010.
Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting  Warfighter Requirements on Time. GAO-10-382. Washington, D.C.:  March 19, 2010.
Joint Strike Fighter: Significant Challenges Remain as DOD Restructures  Program. GAO-10-520T. Washington, D.C.: March 11, 2010.
Joint Strike Fighter: Strong Risk Management Essential as Program  Enters Most Challenging Phase. GAO-09-711T. Washington, D.C.:  May 20, 2009.