F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System
Published: 2016-04-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The F-35 is the most ambitious and expensive weapon system in DOD's history, with sustainment costs comprising the vast majority of DOD's $1.3 trillion cost estimate. Central to F-35 sustainment is ALIS—a complex system supporting operations, mission planning, supply-chain management, maintenance, and other processes. The F-35 program is approaching several key milestones: the Air Force and Navy are to declare the ability to operate and deploy the F-35 in 2016 and 2018 respectively, and full-rate production of the aircraft is to begin in 2019. However, ALIS has experienced developmental issues and schedule delays that have put aircraft availability and flying missions at risk. The National Defense Authorization Act for Fiscal Year 2016 included a provision that GAO review the F-35's ALIS. This report assesses, among other things, the extent to which DOD has (1) a plan to ensure that ALIS is fully functional as key program milestones approach and (2) credibly and accurately estimated ALIS costs. GAO reviewed F-35 program documentation, interviewed officials, and conducted focus groups with ALIS users.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) is aware of risks that could affect the F-35's Autonomic Logistics Information System (ALIS), but does not have a plan to ensure that ALIS is fully functional as key program milestones approach. ALIS users, including pilots and maintainers, in GAO's focus groups identified benefits of the system, such as the incorporation of multiple functions into a single system. However, users also identified several issues that could result in operational and schedule risks. These include the following:
ALIS may not be deployable : ALIS requires server connectivity and the necessary infrastructure to provide power to the system. The Marine Corps, which often deploys to austere locations, declared in July 2015 its ability to operate and deploy the F-35 without conducting deployability tests of ALIS. A newer version of ALIS was put into operation in the summer of 2015, but DOD has not yet completed comprehensive deployability tests.
ALIS does not have redundant infrastructure : ALIS's current design results in all F-35 data produced across the U.S. fleet to be routed to a Central Point of Entry and then to ALIS's main operating unit with no backup system or redundancy. If either of these fail, it could take the entire F-35 fleet offline.
DOD is taking some steps to address these and other risks such as resolving smaller ALIS functionality issues between major software upgrades and considering the procurement of additional ALIS infrastructure but the department is attending to issues on a case-by-case basis. DOD does not have a plan that prioritizes ALIS risks to ensure that the most important are expediently addressed and that DOD has a fully functional ALIS as program milestones draw close. By continuing to respond to issues on a case-by-case basis rather than in a holistic manner, there is no guarantee that DOD will address the highest risks by the start of full-rate production in 2019, and as a result, DOD may encounter further schedule and development delays, which could affect operations and potentially lead to cost increases.
DOD has estimated total ALIS costs to be about $16.7 billion over the F-35's 56-year life cycle, but performing additional analyses and including historical cost data would increase the credibility and accuracy of DOD's estimate. GAO's cost estimating best practices state that cost estimates should include uncertainty analyses to determine the level of uncertainty associated with the estimate in order to be credible. In addition, credible cost estimates should include sensitivity analyses to examine how changes to individual assumptions and inputs affect the estimate as a whole. DOD's guidance does not require the department to perform these analyses for ALIS, and DOD officials stated that they have not done so in part because ALIS constitutes less than 2 percent of the F-35's estimated total sustainment costs. Program officials said that if ALIS is not fully functional, the F-35 could not be operated as frequently as intended, but a DOD-commissioned plan found that schedule slippage and functionality problems with ALIS could lead to $20-100 billion in additional costs. Without uncertainty and sensitivity analyses, it is unclear how ALIS can affect costs. GAO also found that using historical cost data would make DOD's cost estimate more accurate.
section_title: What GAO Recommends
paragraphs: GAO is making four recommendations including that DOD develop a plan to address ALIS risks, and conduct certain analyses and include historical data to improve its ALIS cost estimate. DOD concurred with developing a plan and partially concurred with the cost estimating recommendations, stating that it follows its own guidance. GAO continues to believe the recommendations are valid, as discussed in the report.

section_title: Letter
paragraphs: The Department of Defense (DOD) calls the F-35 Lightning II the  backbone of U.S. air combat superiority for decades to come, as it plans  for the aircraft to replace the legacy tactical fighter fleets of the Air Force,  Navy, and Marine Corps. With estimated total ownership costs of just  under $1.3 trillion over its 56-year life cycle, the F-35 is the most  ambitious and costly program in DOD’s history. The cost to sustain the  aircraft—estimated to be just under $900 billion—is the primary cost  driver for the F-35 program. Central to F-35 sustainment is the Autonomic  Logistics Information System (ALIS)—a complex system that supports  operations, mission planning, supply-chain management, maintenance,  and other processes. A DOD official has described ALIS as the brains of  the aircraft, calling it one of the three major components that make up the  F-35, along with the airframe and engine. However, ALIS has  experienced developmental issues, including system functionality  problems, and schedule delays that have put key performance  parameters, such as aircraft availability and flying missions, at risk. The   F-35 program is approaching several key milestones: the Air Force and  Navy are to declare the ability to operate and deploy the aircraft (“initial  operational capability”) in 2016 and 2018 respectively, and full-rate  production decision of the program is planned for 2019. Recognizing that  a fully functional ALIS is critical to the program’s overall success, in  October 2015, the F-35 executive program officer testified before  Congress that ALIS is one of the most significant technical and schedule  risks to the program.
We have reported on DOD’s acquisition of the F-35 for many years (see  the Related GAO Products section at the end of this report) and more  recently, on F-35 sustainment issues. In September 2014, we found that:  (1) DOD had not fully addressed several issues—including ALIS-related  problems—that posed risks to the long-term affordability and operational  readiness of the program; (2) DOD had not linked established  sustainment affordability targets with military service budgets, meaning  that the targets may not be representative of what the services could  actually afford; and, (3) DOD had weaknesses with respect to a few cost  assumptions and did not include all necessary analyses to make its  sustainment cost estimates fully reliable. With respect to ALIS, we  reported that although DOD had originally planned to have a fully  functional and effective logistics system by 2017—7 years after originally  intended—to ensure operational readiness and availability, DOD did not  have a performance measurement process, with metrics and targets, to  determine and address performance issues with ALIS. We recommended  that DOD establish a performance measurement process for ALIS that  included, but was not limited to, performance metrics and targets that are  based on intended behavior of the system in actual operations and that  tie system performance to user requirements. DOD concurred with our  recommendation and, as of January 2016, was in the process of  developing a performance-measurement process for ALIS.
Citing the importance of a fully functional ALIS to the overall performance  on the F-35, the Subcommittee on Readiness, House Armed Services  Committee, asked us to review the status, costs, and risks associated  with ALIS. Subsequently, the National Defense Authorization Act for  Fiscal Year 2016 included a provision that we report on ALIS, including  an assessment of the capability of the program to address performance  problems within planned resources. This report assesses the extent to  which DOD has (1) a plan to ensure that ALIS is fully functional as key  program milestones approach, (2) credibly and accurately estimated  costs associated with ALIS, and (3) developed a plan to manage ALIS  training for users.
To determine the extent to which DOD has a plan to ensure ALIS is fully  functional by key program milestones—which include the Air Force and  Navy initial operational capability declarations and the start of full-rate  production for the F-35 program—we reviewed the F-35 program’s  sustainment documents and conducted site visits at a nongeneralizable  sample of five F-35 operational and training sites: Eglin Air Force Base,  Marine Corps Air Base Yuma, Luke Air Force Base, Edwards Air Force  Base, and Nellis Air Force Base. We selected these sites in consultation  with service officials to ensure we obtained perspectives from primary  operational and testing sites. For the purposes of this review, we focused  solely on the U.S. F-35 fleet. During these visits, we also held 17  nongeneralizable focus group sessions with a range of ALIS users (e.g.,  maintainers, pilots) from all three services to obtain information on the  operability and deployability of ALIS, and how any ALIS issues may pose  risks to F-35 operations and sustainment. These focus groups also  included contractor personnel responsible for administering ALIS and  providing training at these sites. Approximately a total of 120 people  participated in these focus groups. Additionally, we interviewed key DOD  and contractor officials. We evaluated the information we obtained for  consistency with GAO’s Schedule Assessment Guide and DOD’s System  Engineering Guide for System of Systems that provide guidance and best  practices on how, prior to meeting key milestones, a plan should be  developed to address specific risks that may be associated with major  weapon acquisitions.
To determine the extent to which DOD has credibly and accurately  estimated ALIS costs, we evaluated the reliability of DOD’s estimate of  ALIS costs contained in the most recent F-35 program office estimate of  F-35 operating and support (O&S) costs using characteristics contained  in GAO’s Cost Estimating and Assessment Guide. For the purposes of  this review, we conducted a limited assessment and used two of the four  general characteristics of sound cost estimating included in this guide:  being credible and accurate. We chose to use these two characteristics  because ALIS is not a stand-alone weapon system program with its own  cost estimate, and its projected costs are included in the larger F-35 cost  estimate. Therefore, we only assessed ALIS costs and not an entire life- cycle cost estimate, and determined that it would not be appropriate to  assess whether the estimate was comprehensive or well-documented  because ALIS costs represent one element of the total F-35 cost  estimate—less than 2 percent of projected F-35 sustainment costs. To  determine whether the credible and accurate characteristics were met, we  reviewed cost-estimating documentation, including data sources,  assumptions, and cost models, and we interviewed cost-estimating  officials from the F-35 program office and the Office of the Director for  Cost Assessment and Program Evaluation (CAPE). We also interviewed  other officials from the F-35 program office and service headquarters to  discuss the cost effects of ALIS schedule delays and development issues.  We found the data to not be fully reliable, which we discuss in further  detail in the report and in appendix II.
Finally, to determine the extent to which DOD has developed a plan to  manage ALIS training for users, we reviewed key documentation related  to ALIS and F-35 training, and used information from our focus-group  sessions with a range of ALIS users from all three services to obtain  information on the current state of ALIS training. We also interviewed key  DOD and contractor officials. We evaluated all of the information we  received using GAO-developed and industry best practices for  information-technology training, and guidance on assessing strategic  training and development efforts in the federal government.
We conducted this performance audit from April 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I provides a more  detailed description of our scope and methodology.
subsections: section_title: Background
subsections: section_title: F-35 Program and Costs
paragraphs: The F-35 Lightning II program, also known as the Joint Strike Fighter, is a  joint, multinational acquisition intended to develop and field a family of  next-generation strike fighter aircraft for the United States Air Force,  Navy, and Marine Corps, and eight international partners. According to  DOD, there will be three variants of the F-35:  1.  The conventional takeoff and landing (CTOL) variant, designated the  F-35A, will be a multirole, stealthy strike aircraft replacement for the  Air Force’s F-16 Falcon and the A-10 Thunderbolt II aircraft, and will  complement the F-22A Raptor (see fig. 1).  2.  The short takeoff and vertical landing (STOVL) variant, designated the  F-35B, will be a multirole, stealthy strike fighter that will replace the  Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier aircraft.  3.  The carrier-suitable variant (CV), designated the F-35C, will provide  the Navy a multirole, stealthy strike aircraft to complement the F/A-18  E/F Super Hornet. The Marine Corps will also field a limited number of  F-35C CVs.
Lockheed Martin is the primary aircraft contractor and Pratt & Whitney is  the engine contractor.
Although the acquisition costs of the F-35 program are about $400 billion,  the most significant cost driver for the program is sustainment. The F-35  O&S costs are those incurred from the initial system deployment through  the end of system operations, and include all costs of operating,  maintaining, and supporting the fielded system. The F-35 program office  develops an annual estimate for the O&S costs of maintaining and  supporting the F-35 for 56 years. In its most recent estimate (2014), the  program office estimates that it will cost about $891 billion to sustain the  entire F-35 fleet over its life cycle.
section_title: Autonomic Logistics Information System (ALIS): Primary Logistics Tool for the F-35
paragraphs: The Autonomic Logistics Information System (ALIS) is a system of  systems that serves as the primary logistics tool to support F-35  operations, mission planning, and sustainment. ALIS helps maintainers  manage tasks including aircraft health and diagnostics, supply-chain  management, and necessary maintenance events. Lockheed Martin is  the prime contractor for ALIS and is responsible for developing and  managing the capabilities of the system, as well as developing training  materials for ALIS users.
According to DOD, ALIS will be co-located with F-35 aircraft both at U.S.  military installations and in theater to support missions and assist with  maintenance and resource allocation. ALIS consists of the overarching  system, the applications housed within it, and the some of the network  infrastructure required to provide global integrated and autonomic support  of the F-35 fleet. It comprises both hardware and software. The hardware  consists of three main components:
The Autonomic Logistics Operating Unit (ALOU): The ALOU is the  computer server that all F-35 data ultimately are sent through and it  supports communications with and between the government and the  contractor’s systems.
The Central Point of Entry (CPE): The CPE is configured to provide  software and data distribution for the entire F-35 fleet in the United  States, enables interoperability with national (government) systems at  the country level, and enables ALIS data connectivity between bases.  Each international partner operating F-35 aircraft is expected to have  its own CPE at other locations.
The Standard Operating Unit (SOU): SOUs provide all ALIS  capabilities to support flying, maintenance, and training. They also  provide access to applications to operate and sustain the aircraft.
As of February 2016, there was one operational ALOU and CPE within  the United States. Each F-35 operating and testing site in the United  States has a varying number of SOUs depending on the site’s number of  aircraft and squadrons, and there are two versions: SOU V1 and SOU V2.  The main difference between the two SOUs is that SOU V2 was designed  to better meet participants’ deployability requirements. While SOU V1  was housed in two 1,600-pound server racks, SOU V2 was designed to  have its components fit into transit cases that are two-man portable, each  weighing approximately 200 pounds. DOD is planning to have at least  one SOU accompany each F-35 squadron. The services organize their  squadrons differently but squadron sizes generally range from 10 to 24  aircraft. The F-35 Operational Requirements Document, which originated  in March 2000 and contains the performance and operational parameters  for the concept of the F-35, calls for an incremental development of F-35  capabilities by aircraft software blocks and phased software releases  during the system development and demonstration phase. This is  concurrent to the production and fielding of small volumes of aircraft  during low-rate initial production. ALIS’s software development is  anticipated to be completed after two more of its versions are released,  the first of which will support the Air Force declaring initial operational  capability in summer 2016. The current fielding status of ALIS is  illustrated in figure 2, which also includes the dates for when the next  software upgrade—ALIS Version 2.0.2—is to be introduced at each F-35  site. ALIS 3.0, which is the version expected to meet the requirements  defined in the Operational Requirements Document, is expected to have  completed its development and commence testing by October 2017 in  line with the end of system development and demonstration. The  release is to be fielded under the low-rate initial production contract in  early 2018 and is intended to be fully functional by 2019.
DOD intends for ALIS software capabilities to include: operational  planning, maintenance, supply-chain management, customer-support  services, training, tech data, system security, and external interfaces. As  stated earlier, ALIS is a system of systems—it comprises multiple  software applications that perform specific functions for F-35 maintainers,  pilots, supply personnel, and data analysts. These separate applications  must be integrated within the system, as well as have interconnectivity  with preexisting “legacy” information systems that are used by the  services for other weapons system platforms. ALIS applications are being  developed by the contractor incrementally, with some applications  currently more functional than others. The F-35 program’s original  requirements state that it must include a fully functional and effective  logistics system to ensure operational readiness and availability. As of  December, average aircraft availability for the year—the percentage of F- 35 aircraft capable of performing missions at a given time—was around  50 percent, whereas DOD had expected it to be 60 percent. Figure 3  includes major applications within ALIS, their intended purpose, and the   F-35 program office’s assessment of the functionality status of each  application. As figure 3 shows, according to DOD, most of the  applications within ALIS currently have functionality issues.
ALIS was originally scheduled to be completed for testing in 2010, as  shown in figure 4. However, that same year the program triggered a  Nunn-McCurdy breach, when unit cost growth exceeded critical  thresholds. As a result, the F-35 program (including ALIS) was  rebaselined in 2012, which established a new acquisition baseline, and  Milestone B was recertified after the breach.
The F-35 program is currently approaching the end of the system  development and demonstration phase, during which ALIS has been  developed, built, and tested to verify that operational requirements are  being met. Concurrently, the program is in low-rate initial production. In  addition, DOD and the services are in the process of declaring major  milestones. Specifically, the Marine Corps declared initial operational  capability in July 2015. The Air Force is scheduled to declare initial  operational capability in August 2016, and the Navy is scheduled to do so  in 2018. By 2018, after all services have declared their initial operational  capability, the program, to include ALIS, is expected to reach full  warfighting capability. The F-35 program plans to begin full-rate  production in 2019, and according to DOD officials, any additional  modifications or upgrades with new capabilities to ALIS will be part of the  program’s follow-on modernization. DOD generally requires programs to  have established sustainment and support systems, like ALIS, for the F- 35 by full-rate production. According to the F-35 Operational  Requirements Document, by full-rate production, all variants must be able  to deploy rapidly, sustain high mission reliability, and sustain a high  sortie-generation rate.
section_title: DOD Is Aware of Risks to ALIS Functionality That Users Have Identified, but Does Not Have a Plan to Prioritize and Address Them as Key Program Milestones Approach
paragraphs: DOD is aware of risks that could affect ALIS but does not have a plan to  prioritize and address them in a holistic manner to ensure that ALIS is  fully functional as the F-35 program approaches key milestones— including Air Force and Navy initial operational capability declarations in  2016 and 2018, respectively, and the start of the program’s full-rate  production in 2019. During our focus groups, ALIS users identified some  beneficial aspects of the system. However, they also identified a variety of  concerns, which may result in operational and schedule risks. The   F-35 program office is aware of these issues, but is currently addressing  them on a case-by-case basis.
subsections: section_title: ALIS Users Identified Benefits but Also Key Risks to the System
paragraphs: ALIS users (pilots, maintainers, administrators, and trainers) in the focus  groups we held at five F-35 operational or testing sites identified some  benefits of the system. For example, maintainers and administrators at  three sites stated that they have seen ALIS’s capabilities improve over  time as the system’s software has been upgraded. In addition, pilots and  maintainers at three sites expressed confidence in ALIS’s future  capabilities as the system continues to improve. Maintainers and trainers  at three sites also found ALIS’s design—which incorporates multiple  functions within one single system—helpful for efficiently executing tasks,  when previously they had to work in multiple, separate systems. For  example, trainers at Eglin Air Force Base stated that, with legacy aircraft,  there are separate systems for tasks such as recording maintenance  work and ordering parts. With the F-35, ALIS houses applications for  these tasks within its system, which can be more convenient to users. In  addition, maintainers at two sites stated that, following a recent software  upgrade, the system processes information faster, which improves  maintenance data input capabilities. Maintainers at three sites also told us  that ALIS performs some tasks better than legacy systems. Specifically,  maintainers at Eglin Air Force Base and Nellis Air Force Base explained  that because ALIS stores information electronically, it eliminates the need  for paper-based manuals commonly used on legacy aircraft. For instance,  maintainers at Eglin Air Force Base noted that the technical data that they  use to assist in aircraft repair is now stored electronically within ALIS and  can be updated as necessary, whereas before this information was  contained in multiple paper-based manuals, which were difficult to  efficiently access and keep up-to-date. Most users we spoke with  recognized that ALIS is a system in development and stated that its  immaturity was to be expected at this stage in the program.
However, during our focus-group sessions, ALIS users also identified  several issues, which, if not addressed, could result in operational and  schedule risks. Table 1 summarizes the risks reported by the majority of  participants in our 17 focus groups. DOD is aware of these risks and, as  discussed later, is addressing risks on a case-by-case basis.
The risks identified by ALIS users during our focus group sessions are  discussed in more detail below.
ALIS may not be able to deploy: Pilots, maintainers, and  administrators at three of the five sites we visited are concerned about  ALIS’s ability to deploy and function in forward locations. For example,  users are concerned about the large server size and connectivity  requirements, and whether the system’s infrastructure can maintain  power and withstand a high-temperature environment. The Marine  Corps, which often deploys to austere locations, did not conduct  deployability tests prior to declaring initial operational capability in July  2015. ALIS’s original requirements did not include specific  deployability requirements, so the system’s original hardware design  consisted of large, heavy racks of servers. DOD officials stated that  the Marine Corps subsequently added specific requirements for a  deployable system to meet its expeditionary mission needs. Although  the more deployable version of ALIS was fielded in summer of 2015,  DOD has yet to complete comprehensive deployability testing. In  December 2015, the Marine Corps participated in an exercise at the  Strategic Expeditionary Landing Field near the Marine Corps Air  Ground Combat Center in California (also known as Twentynine  Palms) that included a short-range, domestic deployability test of the  system. According to DOD officials, the results were positive in that  the Marine Corps transported the system to Twentynine Palms from  its Yuma base and set it up within 2 hours; however, this test did not  include long-range, overseas, ship-based, or combat scenarios. Air  Force and Navy officials stated that they plan to conduct deployabilty  tests prior to declaring initial operational capability over the next 2  years; however, these officials expressed concerns over the ability of  ALIS to function in austere environments and in split-squadron  situations that would require multiple deployable ALIS servers.
ALIS does not have redundancy in its infrastructure: ALIS users  at three of the five sites we visited are concerned that a failure in the  system’s current infrastructure could degrade the system and ground  the fleet. Currently, ALIS information, including data from all U.S.   F-35 sites, flows from the Standard Operating Units (SOU) to a single  national Central Point of Entry, and then to the lone Autonomic  Logistics Operating Unit (ALOU). This data flow process has no  back up system for continuity of operations if either of these servers  were to fail. Specifically, squadron leadership at two sites expressed  concern about how the loss of electricity due to weather or other  damaging situations could adversely affect fleet operations if either  the Central Point of Entry or the ALOU went offline. DOD officials told  us that they recognize this issue and, for short-term losses of  connection, ALIS users are able to work offline. Program officials also  said that they are in the early stages of trying to procure up to two  additional ALOUs and possibly relocating the Central Point of Entry to  another F-35 site. However, as of January 2016, DOD officials had  just begun to explore this option and have not allocated any resources  to support the idea.
ALIS does not effectively communicate with legacy aircraft  systems: Maintainers and pilots at three of the five sites we visited  were concerned that ALIS does not have much interoperability with  legacy aircraft systems. For example, while ALIS was designed to  house multiple applications within it, the Air Force, Navy, and Marine  Corps will continue to use legacy systems to operate and maintain  other weapon systems. The ability to share information between ALIS  and these legacy systems is vital due to the way the services operate.  In particular, Marine Corps officials noted that because the service  operates with squadrons that use data from the Navy’s legacy  system and ALIS, it would be beneficial for those two systems to  communicate with one another. DOD officials stated that the services  are responsible for developing the software interface that can take  information from ALIS and translate it so that it can be communicated  to the legacy systems. However, due to the lack of interoperability  between ALIS and legacy systems, users are being forced to track  data outside of ALIS, which, according to maintainers, is inefficient  and could potentially result in data not being populated back into the  system.
Action Request process is inefficient and problematic:  Maintainers at four of the five sites we visited told us that the current  Action Request (AR) process does not allow for the effective  reporting and resolution of F-35 aircraft and ALIS issues. Personnel  use an application within ALIS to submit an AR about any F-35  problems, including those with ALIS itself, to the contractor for triaging  and ultimate resolution. However, these maintainers explained that  the process is too heavily controlled by the contractor and that users  lack visibility of ARs submitted from other F-35 sites or squadrons.  Consequently, ALIS users have to wait for the contractor to conclude  that multiple sites are experiencing a similar issue, instead of being  able to identify common issues across sites and obtain timely  solutions. In addition, maintainers at three sites and administrators at  one site reported that recent ALIS software upgrades have resulted in  the contractor not receiving ARs, with users unaware of this problem  until they followed up on the ARs’ status. DOD officials told us they  are aware of the issues surrounding the AR process and are  collecting information on the types of ARs submitted from all sites.  They stated that the largest types of ARs are related to data quality  and integration management, and that this has been the case for the  last 12 months.
ALIS has data accuracy and accessibility issues: ALIS users at all  five sites we visited are concerned with data accuracy issues within  the system, including missing or inaccurate data and inaccessibility of  raw data within ALIS. Specifically, maintainers frequently have to  resolve error messages for parts linked to electronic equipment logs  that contain missing or inaccurate data when they try to fix a problem  on the aircraft. Maintainers at two sites stated that recurring issues  with electronic equipment logs have caused them to spend significant  time resolving these issues instead of tending to other aircraft  issues. Additionally, they stated that parts requiring scheduled  maintenance are not being tracked or updated accurately in ALIS.  Program officials stated that these are life-limited parts that must be  replaced by a certain time frame to avoid safety risks to the aircraft.  To mitigate this issue, maintainers are currently logging this  information outside of ALIS. Maintainers at Eglin Air Force Base said  that they are spending 13 hours on average every day to track this  information. Finally, maintainers at three sites stated they would like  the ability to access raw data in ALIS to produce service-related  reports. DOD officials stated that ALIS was designed to be used  across services and, as such, reporting tools are not necessarily  service-specific. However, ALIS users that operate the system daily  continue to have issues with accessing the data required to keep  aircraft mission-capable and generating service-specific reports for  their squadrons.
Off-Board Mission Support and Training Management System  applications are immature: Pilots and maintainers across all five  sites we visited are concerned with the maturity and functionality of  ALIS’s Off-Board Mission Support (OMS) and Training Management  System (TMS) applications. OMS is a key application designed for  pilots to conduct essential mission planning and debriefing.  Specifically, pilots at all five sites thought that the OMS application  was poorly designed, cumbersome, and not user-friendly, especially  with providing the necessary information they need to conduct their  missions. Due to OMS’s current lack of functionality, pilots at two  locations stated that they are forced to track vital mission planning  information, information expected to reside within ALIS, outside of the  system. According to the Office of the Director of Test and Evaluation,  OMS’s lack of functionality could have an effect on combat missions  and operational tempo.
The Training Management System (TMS) is designed for pilots and  maintainers to track training qualifications and assign personnel to  carry out specific tasks based on their qualifications. However, pilots  and maintainers at four sites told us that TMS is immature and does  not function as intended. Maintainers at one site explained that TMS  is supposed to keep track of maintainers’ and pilots’ qualifications  and, based on that information, assign proper permission levels and  controls to a qualified maintainer to repair a problem on the aircraft.  Instead, this is currently being tracked outside of ALIS, which is  inefficient and could potentially result in this information not being  populated back into the system.
Security risks exist: ALIS users cited concerns related to the  system’s security. For example, pilots at one location explained that  compact discs have to be recorded to move classified information  from the aircraft into the classified network, rather than the system  transmitting the information automatically—a practice that they said  poses security risks. In addition, the ALOU and Central Point of Entry,  as discussed earlier, are potential single points of failure and could be  a security risk. A 2012 DOD Inspector General’s report on ALIS also  highlighted some security issues, including security accreditation and  testing of hardware. Since that report, the F-35 program office has  formed a team and developed a process to test, validate, and  continuously monitor the security of ALIS applications and their  interfaces with both military information networks and the contractor’s  ALIS architecture.
section_title: DOD Is Addressing ALIS Risks on a Case-by-Case Basis but Does Not Have a Plan to Prioritize and Address Them in a Holistic Manner
paragraphs: DOD is aware of the risks identified by ALIS users, as well as others, and  is addressing some on a case-by-case basis. However, DOD officials  acknowledged that the department does not have a plan that would  prioritize and address key risks in a holistic manner as program  milestones approach. In recent years, the F-35 program has emphasized  the criticality of ALIS to the success of the F-35. In October 2015, the   F-35 Program Executive Officer stated that ALIS is a crucial component  of the F-35 and should be treated as its own weapon system. Additionally,  the Program Executive Officer stated that the program office changed its  organizational structure to provide more senior leadership oversight of  ALIS. Although more focus has been given to ALIS, according to DOD  officials, the F-35 Program Executive Officer has reiterated that the focus  is on completing the current development and testing of ALIS within the  already established time frames, and with the previously planned funding.  As a result, this has created an environment of competing priorities and  limited resources for the entire program in the near term, including ALIS.
ALIS users have identified key risks to ALIS’s functionality that we  highlight in this report, and program office officials acknowledge that  others may exist as well. The F-35 program office has taken some actions  in an attempt to address smaller ALIS functionality issues between major  software upgrades, and is considering the procurement of additional ALIS  servers to add redundancy to the system. However, the current approach  does not prioritize issues in a way that clearly designates which issues  must be addressed within the time left in the system development and  demonstration phase, and which issues could be addressed later as part  of follow-on modernization. GAO guidance and DOD best practices  emphasize that, prior to meeting key milestones, a plan to address  specific risks that may be associated with major weapon acquisitions  should be developed. Specifically, GAO’s Schedule Assessment Guide  states that a high-quality and reliable acquisition schedule includes the  need to plan for and address major risks prior to meeting milestone  dates. DOD’s System Engineering Guide for System of Systems  includes risk management as a key aspect of system engineering. It helps  to ensure that program costs, schedules, and performance objectives are  achieved at every stage in the life cycle and helps to communicate to all  stakeholders the process for uncovering, determining the scope of, and  managing program uncertainties. Although key milestones—such as Air  Force and Navy initial operational capability declarations and the start of  full-rate production—are quickly approaching, DOD does not have a plan  that prioritizes ALIS risks to ensure that the most important are  expediently addressed and that ALIS is fully functional by these  milestones. Furthermore, by continuing to address issues on a case-by- case basis, DOD risks that its solution to one issue could exacerbate  another—for example, in addressing a security risk in isolation, DOD  could inadvertently create further risks to data accessibility. According to  F-35 program officials, a functional ALIS is key to the operational  capability of the aircraft and the day-to-day ability to sustain the aircraft.  Moreover, the department expects to significantly increase aircraft  production within the next 5 years, so the number of aircraft that must be  maintained and kept ready for flight will soon grow. By continuing to  respond to issues on a case-by-case basis rather than in a holistic  manner, there is no guarantee that DOD will address the highest risks,  and as a result, DOD may encounter further schedule and development  delays, including system upgrades, which could affect operations and  potentially lead to cost increases.
section_title: DOD Has Estimated the Costs of ALIS, but Including Additional Analyses and Information Would Increase the Credibility and Accuracy of Its Estimate
paragraphs: DOD has estimated total ALIS costs to be approximately $16.7 billion  over its 56-year life cycle. However, the estimate is not fully credible since  DOD has not performed uncertainty and sensitivity analyses as part of its  cost-estimating process. Moreover, while DOD has updated its estimate  to be reflective of some program changes, it is also not fully accurate  since DOD did not use historical cost data—both actual data from ALIS  and data from comparable programs—when developing its ALIS  estimate. Finally, other costs such as service customizations of ALIS may  require additional future resources, and manual workarounds to the  system currently require additional labor resources.
subsections: section_title: DOD Estimates that Total ALIS Costs Are about $16.7 Billion over a 56- Year Life Cycle
paragraphs: DOD estimates that total ALIS costs are about $16.7 billion—about $562  million to develop the system, about $1.1 billion to procure hardware and  spare parts, and about $15.1 billion to sustain it in then-year dollars.  DOD had expended approximately $505 million to develop ALIS as of  December 2015, and the department estimates that continued  development will cost an additional $57 million through 2017, which is  when DOD expects ALIS 3.0 will be released for testing. In addition to the  purchase cost for ALIS, DOD estimates that ALIS will cost about $15.1  billion to sustain over a 56-year life cycle. Program officials told us that  ALIS development will be completed within the planned resources, but  that the system will require follow-on modernization and that the program  office is currently planning for those additional costs. Table 2 provides  more detail on ALIS cost elements.
section_title: DOD’s Sustainment Cost Estimate for ALIS Is Not Fully Credible
paragraphs: We found that the program office’s estimate for ALIS sustainment costs  minimally met the best practices for a credible cost estimate largely  because it does not include uncertainty and sensitivity analyses (see app.  II for more information on our assessment of the ALIS cost estimate).  Every cost estimate contains a degree of uncertainty because of the  many assumptions that must be made about the future. To mitigate this  uncertainty, a variety of checks and analyses can be conducted to  determine the credibility of the assumptions and the estimate as a whole.  According to GAO’s Cost Estimating and Assessment Guide, cost  estimates should include uncertainty analyses to determine the level of  uncertainty associated with the estimate in order to be credible. A  quantitative uncertainty analysis can provide a broad overall assessment  of the risk in the cost estimate. In addition, credible cost estimates should  include sensitivity analyses to examine how changes to individual  assumptions and inputs affect the estimate as a whole.
Although ALIS is a multibillion dollar system, it is not a formally  designated stand-alone weapons system program; therefore, DOD is not  required to perform a separate estimate of the system’s projected costs.  ALIS is one cost element within the overall F-35 program, and the  program office estimates that the system will constitute less than 2  percent of the $891.1 billion total sustainment costs of the program. Cost  estimators told us that despite its critical importance in operating and  maintaining the entire F-35 fleet, since ALIS constitutes a small portion of  the total estimate, it is not considered a major cost driver of the program.  Therefore, cost estimators told us DOD’s cost-estimating guidance does  not require them to perform uncertainty or sensitivity analyses for ALIS.  As part of the planning process for F-35 sustainment, program officials  have examined the role of F-35 information systems (which includes  ALIS) in relation to five major value streams—maintenance, supply chain,  training, management, and sustaining engineering. While program  officials said this process helped map the connections and  interdependence between program elements, it did not quantify in  uncertainty or sensitivity analyses the potential effects of ALIS on  sustainment costs specifically.
Additionally, DOD did not perform analyses to determine how further ALIS  schedule delays or functionality issues could affect other F-35 costs. In  lieu of the analyses, program officials stated that they assume that if ALIS  does not perform as planned, aircraft could not be flown as frequently as  intended, and this lower-than-expected utilization rate would therefore  decrease sustainment costs. A 2013 DOD-commissioned study on  reducing F-35 costs found areas of potential cost savings, but the  implementation plan for this study also found that ALIS presents  significant risk to the program. Particularly, this plan found that any  functionality problems or schedule slippage with ALIS will have a  significant impact on costs—with downstream additional costs due to  performance and schedule delays potentially reaching up to $20-100  billion. The different conclusions drawn by the program office and this  study suggest that DOD could better understand the effects that ALIS  issues could have on overall program costs. Without performing  uncertainty or sensitivity analyses, DOD will not understand how  variabilities in ALIS-related assumptions could affect the estimate as a  whole and the potential range of costs resulting from these variabilities.
section_title: DOD’s ALIS Cost Estimate Is Not Fully Accurate, and Upcoming Program Changes May Affect Future Estimates
paragraphs: We also assessed the ALIS estimate for accuracy and found that DOD  partially met the standards for an accurate cost estimate. GAO’s Cost  Estimating and Assessment Guide states a cost estimate should be  based on historical data—both actual costs of the program and those of  comparable programs—which can be used to challenge optimistic  assumptions and bring more realism to a cost estimate. While we found  that the DOD substantially met some best practices for an accurate cost  estimate by properly adjusting for inflation and not including mathematical  errors, the estimate uses contractor-provided data for material costs  instead of actual ALIS costs or historical cost data from analogous  programs that would make the estimate more accurate. Cost estimating  officials said that they did not base their ALIS estimates on historical cost  data because they believe that there are no programs analogous to ALIS.  For example, there is a logistics system for the Air Force’s F-22  program—also a fifth-generation aircraft—but officials stated that is far  less complex than ALIS and does not include all of ALIS’s applications  and intended functions. However, multiple versions of ALIS have been  fielded since 2010 and using historical data on known ALIS costs, as well  as analogous data from the F-22 or other programs, would make the  estimate more accurately representative of likely sustainment costs (see  app. II for more information on our assessment).
GAO’s Cost Estimating and Assessment Guide also states that an  estimate should be updated regularly to reflect significant changes in the  program—such as when schedules or other assumptions change—so  that it is always reflecting current status. The program office updates its  estimate annually and incorporates program changes and evolving  assumptions in these updates, documenting the changes from year to  year. However, the program office does not update all elements of the  cost estimate. For example, technology refresh accounts for  approximately 25 percent of ALIS sustainment costs and program officials  were able to tell us the assumptions used to calculate these costs,  however, they were unable to tell us, or identify within the estimating  model, where these data came from or when their underlying  assumptions were developed. Additionally, over the course of our  review, program officials highlighted some recent or upcoming program  changes, such as the need for additional infrastructure, that were not  included in their last estimate.
The program office and services are exploring ways to decrease F-35  sustainment costs. For example, the services are taking different  approaches to administering and maintaining ALIS, with the Marine Corps  planning on using its own personnel, and the Air Force planning on using  contractor support to administer the system, troubleshoot problems, and  keep the system operating smoothly with software patches and other  continuous improvements. Based on these service plans, the program  office estimates that ALIS administration will cost more than $7 billion  over the F-35 life cycle, in addition to about $1 billion for other contractor  labor needs. Program officials said that these amounts may change  based on upcoming sustainment decisions, including a potential way to  decrease administrative costs by establishing regional centers that could  provide this support to a number of F-35 sites rather than having  contracted administrators at each site.
Other program changes have the potential to increase ALIS costs and  future estimates. Although DOD has included estimated costs of ALIS  technology refresh and software licensing, program officials have stated  that the current estimate does not sufficiently capture the full costs of  follow-on modernization that will be required when upgrades or new  versions of the system’s commercial off-the-shelf software are released.  Because ALIS comprises multiple applications and interfaces with service  networks and legacy information systems, engineers will have to  reintegrate ALIS with other systems as well as the applications within  ALIS in step with continuous upgrades and software improvements.  Although DOD expects ALIS 3.0 to be the fully capable version that  meets program requirements, officials at the program office and from all  three services stressed that ALIS will not be a static system and that  improvements and upgrades will not only be expected, but required, to  keep ahead of technology obsolescence and evolve with emerging threat  environments. Program officials stated that there is a need to build these  follow-on modernization costs into the budgeting and cost-estimating  processes.
Program officials also stated that they may procure up to two more  ALOUs for back-up and necessary redundancy and that, in addition to  addressing the current risk of the single ALOU becoming a single point of  failure, these additional ALOUs may facilitate greater government  operation of ALIS and increase the potential for greater competition of  future sustainment contracts. The program office bases much of its  estimate on inputs from the services, such as their expected personnel  needs and how they plan to operate and sustain their F-35 squadrons.  The program plans to field one SOU for each F-35 squadron, and another  SOU at each forward operating location. Since deploying squadrons do  not normally transit to the forward location together as one unit but rather  in a more staggered fashion, having an SOU at both the squadron’s home  base and another prepositioned at the forward deployed location would  avoid potential problems of having a split squadron sustained with just  one SOU. Having additional SOUs positioned at forward deployed  locations would likely increase procurement costs and the downstream  costs of maintaining and replacing them.
While the program office has been incorporating some program changes  and adjusting its cost estimating assumptions as the F-35 program grows  and evolves, it is important to continue this effort in its annual estimate  updates to reflect current or planned program changes such as those  described above. Additionally, as the program gains more experience  operating ALIS, using historical data—especially actual program costs as  they become available—as the basis for its estimates can result in a more  accurate and realistic picture of ALIS costs. Unless DOD’s estimate is  based on an assessment of the most likely costs, the estimate may not be  representative of how much it will cost to sustain ALIS and may inhibit  informed decision making.
section_title: Service Customizations and Workarounds Used to Overcome ALIS Limitations Could Require Additional Resources
paragraphs: There are other ALIS-related costs that are not included in the estimate  but that may place additional financial or manpower burdens on the  services as ALIS continues to be concurrently developed and fielded.  Services are responsible for bearing the costs of engineering any service- specific ALIS customizations. According to DOD officials, ALIS  engineering changes must be agreed-upon by all services and partners in  order to become new requirements. The program office then  communicates these requested changes to the contractor software  engineers developing ALIS. However, as ALIS users become more  familiar with the system and its limitations, the services may request  additional changes to the design of ALIS to meet their specific  airworthiness and reporting requirements, and incur the additional costs  to meet these needs.
Some service-specific reporting requirements are being met through the  use of time-consuming workarounds employed by ALIS users to  compensate for current system limitations. According to program officials,  some service-specific reporting requirements are not addressed by  current ALIS functionality. For example, maintainers at several sites said  that they must request maintenance data entered and housed within ALIS  from the contractor because they cannot access this information  independently. They then use these raw data to generate reports on  aircraft status using software programs outside of ALIS because the  system does not have the capability to extract and process the data in the  form that the services require. Program officials told us that they expect  some of these workarounds to become unnecessary after upcoming  system improvements. Other ALIS functionality issues that create the  need for workarounds do not have scheduled solutions, so both program  and service headquarters officials are in the process of examining the use  of these workarounds to determine whether they are truly necessary and  will require an ALIS design change or whether service expectations  should be better managed. Program officials told us that this workaround  is not a problem with ALIS itself, but more an issue of the services  requiring reports that the system was not designed to provide. Users at all  the sites we visited told us that ALIS should have the functionality to  create reports that they need.
Program officials said that there is not an extra financial cost associated  with these workarounds since they are performed by service personnel,  but rather there is an opportunity cost to performing them—the additional  time personnel spend on manually analyzing data and generating reports  rather than performing their primary job duties. Neither the program office  nor the services track the use of ALIS workarounds across all F-35  squadrons, but squadron leadership at Eglin Air Force Base provided  examples of workarounds performed at their site including the estimated  amount of time personnel must spend to overcome ALIS shortcomings.  They estimate that personnel at that location alone spend approximately  150 man-hours per week on these workarounds—nearly the equivalent of  having four full-time personnel dedicated to manually creating reports  because, according to officials, ALIS currently cannot. Eglin Air Force  Base personnel have created and updated 14 manual products mainly  because of ALIS report limitations and because ALIS does not track  information needed by senior leaders for unit-level analysis and  subsequent maintenance and logistics decisions. They added that without  these manual products, analysis, scheduling, and maintenance  operations would suffer degradation and that mission accomplishment  could be jeopardized.
The use of workarounds varied across the F-35 sites we visited, and  officials said that Eglin Air Force Base’s examples are not necessarily  representative of other sites’ workarounds. They provide a sense of scale,  however, for the labor burden placed on personnel because of current  system immaturity and functionality issues. Personnel at F-35 sites we  visited told us that the extra time they spend on ALIS workarounds  detracts from the time they have to maintain proficiency in their  specialties, prevents them from coaching and training their subordinates,  and decreases the amount of time they have to perform collateral duties.  They added that ALIS is not yet fully autonomic and will require significant  additional system improvements for it to perform to their expectations.
The use of these ALIS workarounds highlights the current immature state  of the system and underscores the need for DOD to prioritize, and  address key ALIS risks as discussed earlier in this report. As the F-35  program increases production over the next several years, sustained  attention to addressing these issues and improving estimates of ALIS  costs can help decision makers better direct resources and ensure that  ALIS meets the needs of its users.
section_title: DOD Does Not Have a Program-Wide Training Plan for ALIS Users but Has Taken Some Initial Steps to Address Training Shortfalls
paragraphs: DOD does not have a program-wide training plan for ALIS, but has taken  initial steps to address training shortfalls. According to ALIS users at all  five F-35 operational and training sites where we conducted our focus  groups, training for ALIS is ineffective, and lacks a standardized, common  curriculum for teaching users how to operate ALIS. Basic ALIS training  courses are made available to all ALIS users at the F-35 Academic  Training Center at Eglin Air Force Base. The training, according to ALIS  users across all five sites, consists of a series of PowerPoint slides that  are geared toward illustrating the conceptual nature of the system— showing how all the applications within ALIS are supposed to work— rather than how to actually operate the system as it currently exists.  Furthermore, several ALIS users described the training as ineffective  because it does not teach users how to explicitly operate specific ALIS  applications. For example, the slideshow-based classroom training, which  is optional for ALIS users, is not customized for the different users of  ALIS. Pilots, maintainers, supply personnel, data analysts, and other F-35  specialists are required to use certain ALIS applications, and often in  different ways based on their job requirements. Maintainers at two of the  five sites said that the classroom training reputation has become so bad  that many new maintainers choose to skip the courses and proceed  directly to on-the-job training when they begin working on the F-35.
From the outset of the original ALIS release in 2010, DOD has relied on  the contractor to manage all ALIS training. ALIS training is currently  heavily dependent on learning on the job, which consists of users learning  how to operate ALIS from colleagues and through trial and error.  According to DOD officials, this is not an uncommon practice for new  systems within the department; however, those officials agree that  training typically begins with a common classroom curriculum, and is  supplemented with on-the-job-training. Almost every ALIS user in our  focus groups noted that they do not learn how to operate any ALIS  applications until on-the-job-training begins on the flight line. Specifically,  the classroom training does not afford an opportunity to practice on  “sandbox” or “ghost” systems that would simulate how ALIS is used on a  daily basis by users. Instead, the practice comes in a live operational  environment, where basic ALIS functions and practices are taught by  supervisors or other users within the respective squadron. According to  ALIS users at all five sites, this practice has led to users learning  inconsistent methods and shortcuts for maneuvering through the different  ALIS applications. In most cases, these practices differ between  squadrons and services; however, there were also cases that some  maintainers highlighted where users were learning different practices  within the same squadron. ALIS users reported that they have created  different workarounds to overcome ALIS functionality issues, but have  also cited instances of not using ALIS as intended because the system is  unwieldy and time-consuming. ALIS users said they are learning to  operate the system in different ways and then perpetuating these  methods, creating a situation where ALIS may not be operated in the  most effective, efficient, or up-to-date way across all F-35 sites.  Furthermore, they are learning to use the system in a live operational  environment, running the risk of making errors that could ultimately affect  aircraft availability.
Program officials acknowledged the training shortfalls identified in the  focus-group sessions. In response to these shortfalls, the program office  has taken some initial steps to address some of the issues. Specifically,  the program office, in concert with the contractor, has developed Mobile  Training Teams to offer a way to train ALIS users outside of the F-35  Academic Training Center and at their specific F-35 sites. These teams  are deployed to F-35 operational and testing sites to help keep ALIS  users up-to-date with ALIS software version releases, which, according to  ALIS users at all sites we visited, had been a significant problem with  ALIS training. Specifically, according to ALIS users at all these sites, ALIS  trainers and administrators are rarely up-to-date on the latest ALIS  releases and functionality changes; therefore, it has led to inconsistencies  in teachings and practices at the squadron level. Mobile Training Teams  offer ALIS version-specific training at each site based on a sequencing  schedule developed by the program office.
In addition, the program office has rolled out an ALIS Training Evaluation  to determine the current state of ALIS training across all F-35 sites. The  end goal of this evaluative process is to identify underlying training  deficiencies, and to develop corrective courses of action to mitigate these  deficiencies. The process will include a series of site visits to include  course audits, interviews, curricula inspections, and stakeholder surveys  as applicable for root-cause analysis. As of January 2016, a team within  the program office had just begun this process; therefore, they could not  provide any details beyond the effort’s scope, methodology, and  associated time frames.
According to best practices of information-technology training, effective  training of users is essential to the workforce supporting an information- technology system. The practices suggest that entities develop  Strategic Learning Plans, or Overarching Training Plans, to help align  training programs with priorities. Furthermore, as part of this process, the  practices state that it is important that the training design and delivery  process ensures learning occurs during the training and also ensures that  the user applies the training on the job. Additional guidance on strategic  training in the federal government states that training plans can aid in the  performance of government programs. Specifically, a training plan can  present a business case for proposed training and development  investments, including the identified problem or opportunity, the concept  for an improved situation or condition, and linkages with strategic  objectives.
According to DOD officials, ALIS training has been a difficult process to  manage because of the dynamic nature in which ALIS has been  developed and upgraded since its initial release. Because new versions  of ALIS have been regularly released in a staggered manner across   F-35 sites, they said it has been difficult to sufficiently train all ALIS users  on the most up-to-date versions and teach consistent practices. However,  with only one major version upgrade remaining prior to version 3.0 (the  final major software release), issues related to constantly changing the  system should decrease. A standardized, program-wide training plan  could remove the emphasis from on-the-job training and provide a  comprehensive, standardized training curriculum across the program.  Without a program-wide training plan that assures that consistent learning  occurs in the classroom, and is then applied by users on the job, the  program runs the risk of continuing to allow users to learn irregular or  incorrect practices through a training culture driven by on-the-job training,  which could impact aircraft availability and safety.
section_title: Conclusions
paragraphs: The F-35 program is DOD’s largest and most costly acquisition program  to date. According to senior defense officials, not only does the F-35  program represent the future of tactical fighter aircraft in our military, but it  is also vital to the security of our nation moving forward. With the Marine  Corps already having declared the F-35 both deployable and combat- ready, the Air Force and Navy set to do the same within the next 2 years,  and the full-rate production of the aircraft set for 2019, including the ramp  up of sustainment activities, it is imperative that DOD address major risks  associated with its central logistics system—ALIS. The program office has  taken steps to identify risks associated with ALIS, including all of those  identified by participants in our focus groups—and has begun, on a case- by-case basis, to address some of these risks; however, without a plan to  prioritize risks and address them in a systematic and holistic manner,  DOD runs the risk of having an ALIS that is not fully functional as it  approaches key program milestones. Without a fully functional ALIS,  DOD could face operational and schedule risks and potential cost  increases to a program that is already the most expensive in DOD’s  history.
Although DOD has estimated the costs of ALIS, additional information  would increase the accuracy and credibility of the estimate. While ALIS is  projected to constitute less than 2 percent of the $891.1 billion total  sustainment costs of the program, the financial impact that nonfunctional  aspects of ALIS may have on the overall operations and sustainment of  the aircraft could be significant. Until DOD does more analyses to  determine the impact ALIS has (and the impact a nonfunctional ALIS  could have) on overall sustainment costs, DOD will not know how much  the costs of ALIS and overall sustainment could fluctuate. Furthermore,  while the program office has been incorporating some program changes  and adjusting its cost-estimating assumptions for ALIS, it is important for  the program office to improve the reliability of its estimate by using  historical data. It is also important for DOD to incorporate program  changes that will likely affect ALIS costs in future estimates, such as  decisions to enhance ALIS infrastructure or decrease planned numbers of  administrative personnel with regional support centers.
Finally, although DOD has recognized that ALIS training needs  improvement and has made some temporary fixes to address the current  shortcomings, DOD has yet to develop a program-wide training plan that  would take the focus off an almost explicit on-the-job-training approach,  and provide greater consistency among ALIS users. Considering the  importance ALIS has to the operations and sustainment of the aircraft,  and that DOD plans to purchase and operate nearly 500 more F-35s in  the next 5 years, it will be important that ALIS users be on the same page  with regard to operating the system. Further training inconsistencies could  lead to impacts on aircraft availability and safety.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the F-35 Program  Executive Officer to take the following four actions:
To ensure that risks associated with ALIS are addressed expediently  and holistically, develop a plan that would prioritize and address ALIS  issues, prior to the start of full-rate production for the program.
To improve the reliability of its cost estimates, conduct uncertainty and  sensitivity analyses consistent with cost-estimating best practices  identified in GAO’s Cost Estimating and Assessment Guide.
To improve the reliability of its cost estimates, ensure that future  estimates of ALIS costs use historical data as available and reflect  significant program changes consistent with cost-estimating best  practices identified in GAO’s Cost Estimating and Assessment Guide.
To ensure that ALIS training issues are fully addressed, develop a  standardized, program-wide plan for ALIS training through the life  cycle of the program.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with two  recommendations and partially concurred with two recommendations.  DOD’s comments are summarized below and reprinted in appendix III.  DOD also provided technical comments, which we have incorporated into  our report where appropriate.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to develop a plan that would  prioritize and address ALIS issues, prior to the start of full-rate production  for the program. DOD stated that the F-35 program office began  developing an ALIS Technical Roadmap in early 2016. The department  added that at its completion later in 2016, this roadmap will be the  foundation of a plan to identify, document, and prioritize ALIS risks;  address them holistically; and inform budget priorities, as the program  transitions from development into sustainment and follow-on  modernization. Additionally, to mitigate the risk of single-point failures in  the infrastructure, the program office contracted to acquire backup ALIS  hardware in 2015; the backup hardware will be operational by early 2017.  We state in our report that the department was aware of the risks to ALIS  and believe that if DOD develops a plan to identify, document, and  prioritize ALIS risks, address them holistically, and inform budget priorities  prior to full-rate production, this action should address our  recommendation. We further state in our report that DOD was in the early  stages of acquiring backup ALIS hardware to mitigate the risk of single- point failures in the infrastructure, but had not yet allotted funding. We  believe this backup system will be critical as the program approaches full- rate production.
DOD partially concurred with the recommendation that the Secretary of  Defense direct the F-35 Program Executive Officer to conduct uncertainty  and sensitivity analyses consistent with cost-estimating best practices  identified in GAO’s Cost Estimating and Assessment Guide. DOD stated  that the department considers the sensitivity analyses that the F-35  program office performs to be a form of uncertainty analysis, as described  in DOD’s Cost Assessment and Program Evaluation Operating & Support  Cost Estimating Guide; however, the DOD cost estimating guidance does  not require DOD to conduct a sensitivity or uncertainty analysis on ALIS  since DOD does not consider ALIS a major cost driver of the F-35  program. As our report states, according to GAO’s Cost Estimating and  Assessment Guide, cost estimates should include uncertainty analyses to  determine the level of uncertainty associated with the estimate in order to  be credible. A quantitative uncertainty analysis can provide a broad  overall assessment of the risk in the cost estimate. A sensitivity analysis  can examine how changes to individual assumptions and inputs affect the  estimate as a whole. Although the F-35 program office may conduct  analyses consistent with DOD cost estimating guidance, it has not  conducted an uncertainty or sensitivity analysis specifically for ALIS.  Although ALIS is projected to constitute less than 2 percent of the $891.1  billion total sustainment costs of the program, the financial impact that  nonfunctional aspects of ALIS may have on the overall operations and  sustainment of the aircraft could be significant. For example, a 2013  DOD-commissioned plan found that any functionality problems or  schedule slippage with ALIS will have a significant impact on costs—with  downstream additional costs due to performance and schedule delays  potentially reaching up to $20-100 billion. We continue to believe that  without completing uncertainty and sensitivity analyses to determine the  effect ALIS has (and the impact a nonfunctional ALIS could have) on  overall sustainment costs, DOD will not know how much the costs of ALIS  and overall sustainment could fluctuate.
DOD partially concurred with the recommendation that the Secretary of  Defense direct the F-35 Program Executive Officer to ensure that future  estimates of ALIS costs use historical data as available and reflect  significant program changes consistent with cost-estimating best  practices identified in GAO’s Cost Estimating and Assessment Guide.   DOD stated that the department will ensure that the future F-35 program  ALIS cost estimates continue to use the latest available historical cost  data as appropriate and reflect the latest approved technical baseline  when the program office incorporates these into the program of record,  according to DOD’s Cost Assessment and Program Evaluation Operating  and Support Cost Estimating Guide; however, we found that DOD was  not using all available historical data. As our report states, according to  GAO’s Cost Estimating and Assessment Guide, a cost estimate should  be based on historical data—both actual costs of the program and those  of comparable programs—which can be used to challenge optimistic  assumptions and bring more realism to a cost estimate. While we found  that DOD substantially met some best practices for an accurate cost  estimate by properly adjusting for inflation and not including mathematical  errors, the estimate uses contractor-provided data for material costs  instead of actual ALIS costs or historical cost data from analogous  programs that would make the estimate more accurate. We continue to  believe that it is important for the program to improve the reliability of its  ALIS estimate by using historical data to the greatest extent possible. It is  also important for DOD to incorporate program changes that will likely  affect ALIS costs in future estimates, such as decisions to enhance ALIS  infrastructure or decrease planned numbers of administrative personnel  for regional support centers.
DOD concurred with the recommendation that the Secretary of Defense  direct the F-35 Program Executive Officer to develop a standardized,  program-wide plan for ALIS training through the life cycle of the program.   DOD stated that, to address immediate issues, the F-35 program office  deployed Mobile Training Teams to assist ALIS users at their home base  locations, and to address longer-term issues, the program office began a  comprehensive evaluation of ALIS training in 2015. According to DOD,  the completion of this evaluation in 2016 will inform development of a plan  to address long-term ALIS training issues. We agree and state in our  report that the F-35 program has taken some positive steps to address  short-term training shortfalls by deploying Mobile Training Teams as a  way to train ALIS users outside of the F-35 Academic Training Center.  We also report that the program has recently begun a comprehensive  evaluation of ALIS training to determine the current state of ALIS training  across all F-35 sites. If the F-35 program leverages this comprehensive  evaluation, when it is completed, to develop a program-wide plan for ALIS  training through the life cycle of the program, this action should address  our recommendation.
We are sending copies of this report to appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Air Force,  Army, and Navy; and the Commandant of the Marine Corps. In addition,  the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff has any questions about this report, please contact me  at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Staff members making key contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which the Department of Defense (DOD) has a  plan to ensure the Autonomic Logistics Information System (ALIS) is fully  functional as the key F-35 program milestones approach, we reviewed  documentation of program plans with relevant sustainment elements  including the F-35 Global Sustainment Plan, the Weapon System  Planning Document, the F-35 Autonomic Logistics Global Sustainment  Concept of Operations, and the F-35 Operational Requirements  Document. We also selected and conducted site visits at a  nongeneralizable sample of five F-35 operational and training sites:
Eglin Air Force Base, Florida
Luke Air Force Base, Arizona
Edwards Air Force Base, California
Nellis Air Force Base, Nevada
Marine Corps Air Base Yuma, Arizona  We selected these sites in consultation with service officials to ensure we  obtained perspectives across all three services and at both operational  and testing sites. During these visits, we convened 17 non-generalizable  focus-group sessions with a range of ALIS users from all three services to  obtain information on the operability and deployability of ALIS, and how  any ALIS issues may pose risks for F-35 operations and sustainment.  Specifically, we convened groups of maintainers, pilots, system  administrators, and trainers. We also held focus groups with contractor  personnel responsible for training and administering ALIS at these sites.  There were approximately a total of 120 participants in these focus  groups. Table 3 includes a breakdown of the focus groups we held at the  various locations.
We worked with our methodologist to develop a focus group script that  included questions across four main categories: Training, ALIS Positives,  ALIS Negatives, and Risks, that was used at all five site visits. For  consistency, our methodologist facilitated all focus group sessions. To  analyze the focus-group responses, we conducted content analyses,  developing categories and sub-categories and coding comments from  each focus group to these categories. After each comment had been  coded by an analyst, another analyst independently reviewed each code  and either agreed or disagreed with the coding decision. Where there was  disagreement in the coding decision, the two analysts discussed it and  came to a resolution. Based on the content analysis, we described the  overarching ALIS benefits and risks obtained from ALIS users about the  system and any risks it poses to the F-35 program as it approaches key  milestones. After obtaining information on DOD’s current approach to  addressing the functionality issues that ALIS users identified, we  evaluated information we obtained for consistency with best practices  from GAO’s Schedule Assessment Guide and DOD’s System  Engineering Guide for System of Systems that provide guidance and best  practices on how, prior to meeting key milestones, a plan to address  specific risks that may be associated with major weapon acquisitions  should be developed. We also interviewed key DOD and contractor  officials to collect information about building and testing ALIS, the  capabilities of ALIS, metrics collected on ALIS’s development and  performance, software upgrades to ALIS, and how the   F-35 program office is addressing ALIS functionality issues.
To determine the extent to which DOD has credibly and accurately  estimated ALIS costs, we evaluated the reliability of DOD’s estimate of  ALIS costs contained in the F-35 program office’s 2014 estimate of F-35  operating and support (O&S) costs, the most up-to-date estimate  completed at the time of our review. The program office completed the  2014 cost estimate in the spring of 2015 and plans to release its 2015  updated estimate in spring of 2016. The Office of the Director for Cost  Assessment and Program Evaluation (CAPE) also performed an F-35  O&S cost estimate in 2013, but has not updated it. Both CAPE and Joint  Program Office officials told us they used the same cost inputs,  methodology, and ground rules and assumptions when estimating ALIS  costs in their respective estimates. We used characteristics contained in  GAO’s Cost Estimating and Assessment Guide to assess the reliability of  DOD’s estimate of ALIS costs. According to the guide, there are four  general characteristics of sound cost estimating: being well-documented,  comprehensive, credible, and accurate. For the purposes of this review,  we conducted a limited assessment and used two of the four general  characteristics of sound cost estimating included in this guide: being  credible and accurate. We chose these characteristics because ALIS  costs represent only one element of the total F-35 cost estimate—less  than 2 percent of projected F-35 sustainment costs—and therefore we  determined that it would not be appropriate to assess whether the  estimate was comprehensive or well-documented. To determine whether  the credible and accurate characteristics were met, we reviewed  documentation used to generate the program office’s estimate, including  data sources, assumptions, and cost models, and we interviewed cost- estimating officials from the program office and CAPE. We also  interviewed other officials from the program office and service  headquarters to discuss the cost effects of ALIS schedule delays and  development issues. Results of our assessment of the estimate’s  credibility and accuracy, along with descriptions of these characteristics  and their associated best practices, are detailed in appendix II of this  report. We found the data to not be fully reliable, which we discuss in  further detail in the report and in appendix II.
Finally, to determine the extent to which DOD has developed a plan to  manage ALIS training for users, we reviewed key documentation related  to ALIS and F-35 training, and used information from our focus-group  sessions across various types of ALIS users from all three services to  obtain information on the current state of ALIS training. We also  interviewed key DOD and contractor officials. We evaluated all of the  information we received using GAO-developed and industry best  practices for information-technology training, and DOD’s Policies and  Procedures for Acquisition of Information Technology.
To address all of our objectives, we collected and analyzed information  and interviewed officials from the following Department of Defense (DOD)  offices:
Office of the Under Secretary of Defense (Acquisitions, Technology  and Logistics)
Office of the Director for Cost Assessment and Program Evaluation  (CAPE)
Office of the Director for Operational Test and Evaluation (DOT&E)
Department of the Air Force
Department of the Navy
Headquarters Marine Corps
F-35 Joint Program Office  We also collected and analyzed information and interviewed officials from  Lockheed Martin in Fort Worth, Texas, and Orlando, Florida.
We conducted this performance audit from April 2015 to April 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides and reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: GAO Scoring of the F-35 Program Office’s 2014 Autonomic Logistics Information System (ALIS) Operating and Support (O&S) Cost Estimate
paragraphs: We completed an assessment of the ALIS costs in the F-35 program  office overall F-35 O&S cost estimate on the basis of two  characteristics—being credible and accurate—and their associated best  practices derived from the GAO Cost Estimating and Assessment Guide.  After reviewing documentation that the program office submitted for its  2014 F-35 ALIS O&S cost estimate, conducting interviews with program  office cost-estimating officials, and reviewing relevant sources, we  determined that these cost estimates are not fully reliable. While we found  that the program office estimate for ALIS is partially accurate, the  estimate is minimally credible. These evaluations are shown in table 4.  We determined the overall assessment rating by assigning each  individual best practice rating a number: Not Met = 1, Minimally Met = 2,  Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the  average of the individual best practice assessment ratings to determine  the overall rating for each of the two characteristics. The resulting  average becomes the Overall Assessment as follows: Not Met = 1.0 to  1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially  Met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered  reliable if the overall assessment ratings for each of the two  characteristics are substantially or fully met. If any of the characteristics  are not met, minimally met, or partially met, then the cost estimate does  not fully reflect the characteristics of a high-quality estimate and cannot  be considered reliable.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition the contact name above, the following staff members made  key contributions to this report: Alissa Czyz, Assistant Director; Steven  Banovac; Jeffrey Hubbard; Jason Lee; Jennie Leotta; Terry Richardson;  Amie Lesser; Alyssa Weir; and Delia Zee.
section_title: Related GAO Products
paragraphs: F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans  to Begin Development of New Capabilities. GAO-16-390. Washington,  D.C.: April 14, 2016.
F-35 Joint Strike Fighter: Assessment Needed to Address Affordability  Challenges. GAO-15-364. Washington, D.C.: April 14, 2015.
F-35 Sustainment: Need for Affordable Strategy, Greater Attention to  Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.:  September 23, 2014.
F-35 Joint Strike Fighter: Slower Than Expected Progress in Software  Testing May Limit Initial Warfighting Capabilities. GAO-14-468T.  Washington, D.C.: March 26 2014.
F-35 Joint Strike Fighter: Problems Completing Software Testing May  Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322.  Washington, D.C.: March 24, 2014.
F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but  Affordability Challenges and Other Risks Remain. GAO-13-690T.  Washington, D.C.: June 19, 2013.
F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but  Affordability Challenges and Other Risks Remain. GAO-13-500T.  Washington, D.C.: April 17, 2013.
F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term  Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March  11, 2013.
Fighter Aircraft: Better Cost Estimates Needed for Extending the Service  Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.:  November 15, 2012  Joint Strike Fighter: DOD Actions Needed to Further Enhance  Restructuring and Address Affordability Risks. GAO-12-437. Washington,  D.C.: June 14, 2012.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-12-400SP. Washington, D.C.: March 29, 2012.
Joint Strike Fighter: Restructuring Added Resources and Reduced Risk,  but Concurrency Is Still a Major Concern. GAO-12-525T. Washington,  D.C.: March 20, 2012.
Joint Strike Fighter: Implications of Program Restructuring and Other  Recent Developments on Key Aspects of DOD’s Prior Alternate Engine  Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19,  2011.
Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.
Joint Strike Fighter: Restructuring Should Improve Outcomes, but  Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.:  March 15, 2011.
Tactical Aircraft: Air Force Fighter Force Structure Reports Generally  Addressed Congressional Mandates, but Reflected Dated Plans and  Guidance, and Limited Analyses. GAO-11-323R. Washington, D.C.:  February 24, 2011.
Defense Management: DOD Needs to Monitor and Assess Corrective  Actions Resulting from Its Corrosion Study of the F-35 Joint Strike  Fighter. GAO-11-171R. Washington D.C.: December 16, 2010.
Joint Strike Fighter: Assessment of DOD’s Funding Projection for the  F136 Alternate Engine. GAO-10-1020R. Washington, D.C.: September  15, 2010.
Tactical Aircraft: DOD’s Ability to Meet Future Requirements Is Uncertain,  with Key Analyses Needed to Inform Upcoming Investment Decisions.  GAO-10-789. Washington, D.C.: July 29, 2010.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-10-388SP. Washington, D.C.: March 30, 2010.
Joint Strike Fighter: Significant Challenges and Decisions Ahead.   GAO-10-478T. Washington, D.C.: March 24, 2010.
Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting  Warfighter Requirements on Time. GAO-10-382. Washington, D.C.:  March 19, 2010.
Joint Strike Fighter: Significant Challenges Remain as DOD Restructures  Program. GAO-10-520T. Washington, D.C.: March 11, 2010.
Joint Strike Fighter: Strong Risk Management Essential as Program  Enters Most Challenging Phase. GAO-09-711T. Washington, D.C.: May  20, 2009.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-09-326SP. Washington, D.C.: March 30, 2009.
Joint Strike Fighter: Accelerating Procurement before Completing  Development Increases the Government’s Financial Risk. GAO-09-303.  Washington D.C.: March 12, 2009.
Defense Acquisitions: Better Weapon Program Outcomes Require  Discipline, Accountability, and Fundamental Changes in the Acquisition  Environment. GAO-08-782T. Washington, D.C.: June 3, 2008.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-08-467SP. Washington, D.C.: March 31, 2008.
Joint Strike Fighter: Impact of Recent Decisions on Program Risks.   GAO-08-569T. Washington, D.C.: March 11, 2008.
Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks.  GAO-08-388. Washington, D.C.: March 11, 2008.
Tactical Aircraft: DOD Needs a Joint and Integrated Investment Strategy.  GAO-07-415. Washington, D.C.: April 2, 2007.
Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter Engine  Program. GAO-07-656T. Washington, D.C.: March 22, 2007.
Joint Strike Fighter: Progress Made and Challenges Remain.   GAO-07-360. Washington, D.C.: March 15, 2007.
Tactical Aircraft: DOD’s Cancellation of the Joint Strike Fighter Alternate  Engine Program Was Not Based on a Comprehensive Analysis.   GAO-06-717R. Washington, D.C.: May 22, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
Defense Acquisitions: Actions Needed to Get Better Results on Weapons  Systems Investments. GAO-06-585T. Washington, D.C.: April 5, 2006.
Tactical Aircraft: Recapitalization Goals Are Not Supported by  Knowledge-Based F-22A and JSF Business Cases. GAO-06-487T.  Washington, D.C.: March 16, 2006.
Joint Strike Fighter: DOD Plans to Enter Production before Testing  Demonstrates Acceptable Performance. GAO-06-356. Washington, D.C.:  March 15, 2006.
Joint Strike Fighter: Management of the Technology Transfer Process.  GAO-06-364. Washington, D.C.: March 14, 2006.
Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications for  Tactical Aircraft Modernization. GAO-05-519T. Washington, D.C: April 6,  2005.
Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter  Program with Different Acquisition Strategy. GAO-05-271. Washington,  D.C.: March 15, 2005.