F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System
Released: 2016-04-14T00:00:00

Letter

Background

F-35 Program and Costs

The F-35 Lightning II program, also known as the Joint Strike Fighter, is a joint, multinational acquisition intended to develop and field a family of next-generation strike fighter aircraft for the United States Air Force, Navy, and Marine Corps, and eight international partners. According to DOD, there will be three variants of the F-35: 1. The conventional takeoff and landing (CTOL) variant, designated the F-35A, will be a multirole, stealthy strike aircraft replacement for the Air Force’s F-16 Falcon and the A-10 Thunderbolt II aircraft, and will complement the F-22A Raptor (see fig. 1). 2. The short takeoff and vertical landing (STOVL) variant, designated the F-35B, will be a multirole, stealthy strike fighter that will replace the Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier aircraft. 3. The carrier-suitable variant (CV), designated the F-35C, will provide the Navy a multirole, stealthy strike aircraft to complement the F/A-18 E/F Super Hornet. The Marine Corps will also field a limited number of F-35C CVs.

Lockheed Martin is the primary aircraft contractor and Pratt & Whitney is the engine contractor.

Although the acquisition costs of the F-35 program are about $400 billion, the most significant cost driver for the program is sustainment. The F-35 O&S costs are those incurred from the initial system deployment through the end of system operations, and include all costs of operating, maintaining, and supporting the fielded system. The F-35 program office develops an annual estimate for the O&S costs of maintaining and supporting the F-35 for 56 years. In its most recent estimate (2014), the program office estimates that it will cost about $891 billion to sustain the entire F-35 fleet over its life cycle.

Autonomic Logistics Information System (ALIS): Primary Logistics Tool for the F-35

The Autonomic Logistics Information System (ALIS) is a system of systems that serves as the primary logistics tool to support F-35 operations, mission planning, and sustainment. ALIS helps maintainers manage tasks including aircraft health and diagnostics, supply-chain management, and necessary maintenance events. Lockheed Martin is the prime contractor for ALIS and is responsible for developing and managing the capabilities of the system, as well as developing training materials for ALIS users.

According to DOD, ALIS will be co-located with F-35 aircraft both at U.S. military installations and in theater to support missions and assist with maintenance and resource allocation. ALIS consists of the overarching system, the applications housed within it, and the some of the network infrastructure required to provide global integrated and autonomic support of the F-35 fleet. It comprises both hardware and software. The hardware consists of three main components:

The Autonomic Logistics Operating Unit (ALOU): The ALOU is the computer server that all F-35 data ultimately are sent through and it supports communications with and between the government and the contractor’s systems.

The Central Point of Entry (CPE): The CPE is configured to provide software and data distribution for the entire F-35 fleet in the United States, enables interoperability with national (government) systems at the country level, and enables ALIS data connectivity between bases. Each international partner operating F-35 aircraft is expected to have its own CPE at other locations.

The Standard Operating Unit (SOU): SOUs provide all ALIS capabilities to support flying, maintenance, and training. They also provide access to applications to operate and sustain the aircraft.

As of February 2016, there was one operational ALOU and CPE within the United States. Each F-35 operating and testing site in the United States has a varying number of SOUs depending on the site’s number of aircraft and squadrons, and there are two versions: SOU V1 and SOU V2. The main difference between the two SOUs is that SOU V2 was designed to better meet participants’ deployability requirements. While SOU V1 was housed in two 1,600-pound server racks, SOU V2 was designed to have its components fit into transit cases that are two-man portable, each weighing approximately 200 pounds. DOD is planning to have at least one SOU accompany each F-35 squadron. The services organize their squadrons differently but squadron sizes generally range from 10 to 24 aircraft. The F-35 Operational Requirements Document, which originated in March 2000 and contains the performance and operational parameters for the concept of the F-35, calls for an incremental development of F-35 capabilities by aircraft software blocks and phased software releases during the system development and demonstration phase. This is concurrent to the production and fielding of small volumes of aircraft during low-rate initial production. ALIS’s software development is anticipated to be completed after two more of its versions are released, the first of which will support the Air Force declaring initial operational capability in summer 2016. The current fielding status of ALIS is illustrated in figure 2, which also includes the dates for when the next software upgrade—ALIS Version 2.0.2—is to be introduced at each F-35 site. ALIS 3.0, which is the version expected to meet the requirements defined in the Operational Requirements Document, is expected to have completed its development and commence testing by October 2017 in line with the end of system development and demonstration. The release is to be fielded under the low-rate initial production contract in early 2018 and is intended to be fully functional by 2019.

DOD intends for ALIS software capabilities to include: operational planning, maintenance, supply-chain management, customer-support services, training, tech data, system security, and external interfaces. As stated earlier, ALIS is a system of systems—it comprises multiple software applications that perform specific functions for F-35 maintainers, pilots, supply personnel, and data analysts. These separate applications must be integrated within the system, as well as have interconnectivity with preexisting “legacy” information systems that are used by the services for other weapons system platforms. ALIS applications are being developed by the contractor incrementally, with some applications currently more functional than others. The F-35 program’s original requirements state that it must include a fully functional and effective logistics system to ensure operational readiness and availability. As of December, average aircraft availability for the year—the percentage of F- 35 aircraft capable of performing missions at a given time—was around 50 percent, whereas DOD had expected it to be 60 percent. Figure 3 includes major applications within ALIS, their intended purpose, and the F-35 program office’s assessment of the functionality status of each application. As figure 3 shows, according to DOD, most of the applications within ALIS currently have functionality issues.

ALIS was originally scheduled to be completed for testing in 2010, as shown in figure 4. However, that same year the program triggered a Nunn-McCurdy breach, when unit cost growth exceeded critical thresholds. As a result, the F-35 program (including ALIS) was rebaselined in 2012, which established a new acquisition baseline, and Milestone B was recertified after the breach.

The F-35 program is currently approaching the end of the system development and demonstration phase, during which ALIS has been developed, built, and tested to verify that operational requirements are being met. Concurrently, the program is in low-rate initial production. In addition, DOD and the services are in the process of declaring major milestones. Specifically, the Marine Corps declared initial operational capability in July 2015. The Air Force is scheduled to declare initial operational capability in August 2016, and the Navy is scheduled to do so in 2018. By 2018, after all services have declared their initial operational capability, the program, to include ALIS, is expected to reach full warfighting capability. The F-35 program plans to begin full-rate production in 2019, and according to DOD officials, any additional modifications or upgrades with new capabilities to ALIS will be part of the program’s follow-on modernization. DOD generally requires programs to have established sustainment and support systems, like ALIS, for the F- 35 by full-rate production. According to the F-35 Operational Requirements Document, by full-rate production, all variants must be able to deploy rapidly, sustain high mission reliability, and sustain a high sortie-generation rate.

DOD Is Aware of Risks to ALIS Functionality That Users Have Identified, but Does Not Have a Plan to Prioritize and Address Them as Key Program Milestones Approach

DOD is aware of risks that could affect ALIS but does not have a plan to prioritize and address them in a holistic manner to ensure that ALIS is fully functional as the F-35 program approaches key milestones— including Air Force and Navy initial operational capability declarations in 2016 and 2018, respectively, and the start of the program’s full-rate production in 2019. During our focus groups, ALIS users identified some beneficial aspects of the system. However, they also identified a variety of concerns, which may result in operational and schedule risks. The F-35 program office is aware of these issues, but is currently addressing them on a case-by-case basis.

ALIS Users Identified Benefits but Also Key Risks to the System

ALIS users (pilots, maintainers, administrators, and trainers) in the focus groups we held at five F-35 operational or testing sites identified some benefits of the system. For example, maintainers and administrators at three sites stated that they have seen ALIS’s capabilities improve over time as the system’s software has been upgraded. In addition, pilots and maintainers at three sites expressed confidence in ALIS’s future capabilities as the system continues to improve. Maintainers and trainers at three sites also found ALIS’s design—which incorporates multiple functions within one single system—helpful for efficiently executing tasks, when previously they had to work in multiple, separate systems. For example, trainers at Eglin Air Force Base stated that, with legacy aircraft, there are separate systems for tasks such as recording maintenance work and ordering parts. With the F-35, ALIS houses applications for these tasks within its system, which can be more convenient to users. In addition, maintainers at two sites stated that, following a recent software upgrade, the system processes information faster, which improves maintenance data input capabilities. Maintainers at three sites also told us that ALIS performs some tasks better than legacy systems. Specifically, maintainers at Eglin Air Force Base and Nellis Air Force Base explained that because ALIS stores information electronically, it eliminates the need for paper-based manuals commonly used on legacy aircraft. For instance, maintainers at Eglin Air Force Base noted that the technical data that they use to assist in aircraft repair is now stored electronically within ALIS and can be updated as necessary, whereas before this information was contained in multiple paper-based manuals, which were difficult to efficiently access and keep up-to-date. Most users we spoke with recognized that ALIS is a system in development and stated that its immaturity was to be expected at this stage in the program.

However, during our focus-group sessions, ALIS users also identified several issues, which, if not addressed, could result in operational and schedule risks. Table 1 summarizes the risks reported by the majority of participants in our 17 focus groups. DOD is aware of these risks and, as discussed later, is addressing risks on a case-by-case basis.

The risks identified by ALIS users during our focus group sessions are discussed in more detail below.

ALIS may not be able to deploy: Pilots, maintainers, and administrators at three of the five sites we visited are concerned about ALIS’s ability to deploy and function in forward locations. For example, users are concerned about the large server size and connectivity requirements, and whether the system’s infrastructure can maintain power and withstand a high-temperature environment. The Marine Corps, which often deploys to austere locations, did not conduct deployability tests prior to declaring initial operational capability in July 2015. ALIS’s original requirements did not include specific deployability requirements, so the system’s original hardware design consisted of large, heavy racks of servers. DOD officials stated that the Marine Corps subsequently added specific requirements for a deployable system to meet its expeditionary mission needs. Although the more deployable version of ALIS was fielded in summer of 2015, DOD has yet to complete comprehensive deployability testing. In December 2015, the Marine Corps participated in an exercise at the Strategic Expeditionary Landing Field near the Marine Corps Air Ground Combat Center in California (also known as Twentynine Palms) that included a short-range, domestic deployability test of the system. According to DOD officials, the results were positive in that the Marine Corps transported the system to Twentynine Palms from its Yuma base and set it up within 2 hours; however, this test did not include long-range, overseas, ship-based, or combat scenarios. Air Force and Navy officials stated that they plan to conduct deployabilty tests prior to declaring initial operational capability over the next 2 years; however, these officials expressed concerns over the ability of ALIS to function in austere environments and in split-squadron situations that would require multiple deployable ALIS servers.

ALIS does not have redundancy in its infrastructure: ALIS users at three of the five sites we visited are concerned that a failure in the system’s current infrastructure could degrade the system and ground the fleet. Currently, ALIS information, including data from all U.S. F-35 sites, flows from the Standard Operating Units (SOU) to a single national Central Point of Entry, and then to the lone Autonomic Logistics Operating Unit (ALOU). This data flow process has no back up system for continuity of operations if either of these servers were to fail. Specifically, squadron leadership at two sites expressed concern about how the loss of electricity due to weather or other damaging situations could adversely affect fleet operations if either the Central Point of Entry or the ALOU went offline. DOD officials told us that they recognize this issue and, for short-term losses of connection, ALIS users are able to work offline. Program officials also said that they are in the early stages of trying to procure up to two additional ALOUs and possibly relocating the Central Point of Entry to another F-35 site. However, as of January 2016, DOD officials had just begun to explore this option and have not allocated any resources to support the idea.

ALIS does not effectively communicate with legacy aircraft systems: Maintainers and pilots at three of the five sites we visited were concerned that ALIS does not have much interoperability with legacy aircraft systems. For example, while ALIS was designed to house multiple applications within it, the Air Force, Navy, and Marine Corps will continue to use legacy systems to operate and maintain other weapon systems. The ability to share information between ALIS and these legacy systems is vital due to the way the services operate. In particular, Marine Corps officials noted that because the service operates with squadrons that use data from the Navy’s legacy system and ALIS, it would be beneficial for those two systems to communicate with one another. DOD officials stated that the services are responsible for developing the software interface that can take information from ALIS and translate it so that it can be communicated to the legacy systems. However, due to the lack of interoperability between ALIS and legacy systems, users are being forced to track data outside of ALIS, which, according to maintainers, is inefficient and could potentially result in data not being populated back into the system.

Action Request process is inefficient and problematic: Maintainers at four of the five sites we visited told us that the current Action Request (AR) process does not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues. Personnel use an application within ALIS to submit an AR about any F-35 problems, including those with ALIS itself, to the contractor for triaging and ultimate resolution. However, these maintainers explained that the process is too heavily controlled by the contractor and that users lack visibility of ARs submitted from other F-35 sites or squadrons. Consequently, ALIS users have to wait for the contractor to conclude that multiple sites are experiencing a similar issue, instead of being able to identify common issues across sites and obtain timely solutions. In addition, maintainers at three sites and administrators at one site reported that recent ALIS software upgrades have resulted in the contractor not receiving ARs, with users unaware of this problem until they followed up on the ARs’ status. DOD officials told us they are aware of the issues surrounding the AR process and are collecting information on the types of ARs submitted from all sites. They stated that the largest types of ARs are related to data quality and integration management, and that this has been the case for the last 12 months.

ALIS has data accuracy and accessibility issues: ALIS users at all five sites we visited are concerned with data accuracy issues within the system, including missing or inaccurate data and inaccessibility of raw data within ALIS. Specifically, maintainers frequently have to resolve error messages for parts linked to electronic equipment logs that contain missing or inaccurate data when they try to fix a problem on the aircraft. Maintainers at two sites stated that recurring issues with electronic equipment logs have caused them to spend significant time resolving these issues instead of tending to other aircraft issues. Additionally, they stated that parts requiring scheduled maintenance are not being tracked or updated accurately in ALIS. Program officials stated that these are life-limited parts that must be replaced by a certain time frame to avoid safety risks to the aircraft. To mitigate this issue, maintainers are currently logging this information outside of ALIS. Maintainers at Eglin Air Force Base said that they are spending 13 hours on average every day to track this information. Finally, maintainers at three sites stated they would like the ability to access raw data in ALIS to produce service-related reports. DOD officials stated that ALIS was designed to be used across services and, as such, reporting tools are not necessarily service-specific. However, ALIS users that operate the system daily continue to have issues with accessing the data required to keep aircraft mission-capable and generating service-specific reports for their squadrons.

Off-Board Mission Support and Training Management System applications are immature: Pilots and maintainers across all five sites we visited are concerned with the maturity and functionality of ALIS’s Off-Board Mission Support (OMS) and Training Management System (TMS) applications. OMS is a key application designed for pilots to conduct essential mission planning and debriefing. Specifically, pilots at all five sites thought that the OMS application was poorly designed, cumbersome, and not user-friendly, especially with providing the necessary information they need to conduct their missions. Due to OMS’s current lack of functionality, pilots at two locations stated that they are forced to track vital mission planning information, information expected to reside within ALIS, outside of the system. According to the Office of the Director of Test and Evaluation, OMS’s lack of functionality could have an effect on combat missions and operational tempo.

The Training Management System (TMS) is designed for pilots and maintainers to track training qualifications and assign personnel to carry out specific tasks based on their qualifications. However, pilots and maintainers at four sites told us that TMS is immature and does not function as intended. Maintainers at one site explained that TMS is supposed to keep track of maintainers’ and pilots’ qualifications and, based on that information, assign proper permission levels and controls to a qualified maintainer to repair a problem on the aircraft. Instead, this is currently being tracked outside of ALIS, which is inefficient and could potentially result in this information not being populated back into the system.

Security risks exist: ALIS users cited concerns related to the system’s security. For example, pilots at one location explained that compact discs have to be recorded to move classified information from the aircraft into the classified network, rather than the system transmitting the information automatically—a practice that they said poses security risks. In addition, the ALOU and Central Point of Entry, as discussed earlier, are potential single points of failure and could be a security risk. A 2012 DOD Inspector General’s report on ALIS also highlighted some security issues, including security accreditation and testing of hardware. Since that report, the F-35 program office has formed a team and developed a process to test, validate, and continuously monitor the security of ALIS applications and their interfaces with both military information networks and the contractor’s ALIS architecture.

DOD Is Addressing ALIS Risks on a Case-by-Case Basis but Does Not Have a Plan to Prioritize and Address Them in a Holistic Manner

DOD is aware of the risks identified by ALIS users, as well as others, and is addressing some on a case-by-case basis. However, DOD officials acknowledged that the department does not have a plan that would prioritize and address key risks in a holistic manner as program milestones approach. In recent years, the F-35 program has emphasized the criticality of ALIS to the success of the F-35. In October 2015, the F-35 Program Executive Officer stated that ALIS is a crucial component of the F-35 and should be treated as its own weapon system. Additionally, the Program Executive Officer stated that the program office changed its organizational structure to provide more senior leadership oversight of ALIS. Although more focus has been given to ALIS, according to DOD officials, the F-35 Program Executive Officer has reiterated that the focus is on completing the current development and testing of ALIS within the already established time frames, and with the previously planned funding. As a result, this has created an environment of competing priorities and limited resources for the entire program in the near term, including ALIS.

ALIS users have identified key risks to ALIS’s functionality that we highlight in this report, and program office officials acknowledge that others may exist as well. The F-35 program office has taken some actions in an attempt to address smaller ALIS functionality issues between major software upgrades, and is considering the procurement of additional ALIS servers to add redundancy to the system. However, the current approach does not prioritize issues in a way that clearly designates which issues must be addressed within the time left in the system development and demonstration phase, and which issues could be addressed later as part of follow-on modernization. GAO guidance and DOD best practices emphasize that, prior to meeting key milestones, a plan to address specific risks that may be associated with major weapon acquisitions should be developed. Specifically, GAO’s Schedule Assessment Guide states that a high-quality and reliable acquisition schedule includes the need to plan for and address major risks prior to meeting milestone dates. DOD’s System Engineering Guide for System of Systems includes risk management as a key aspect of system engineering. It helps to ensure that program costs, schedules, and performance objectives are achieved at every stage in the life cycle and helps to communicate to all stakeholders the process for uncovering, determining the scope of, and managing program uncertainties. Although key milestones—such as Air Force and Navy initial operational capability declarations and the start of full-rate production—are quickly approaching, DOD does not have a plan that prioritizes ALIS risks to ensure that the most important are expediently addressed and that ALIS is fully functional by these milestones. Furthermore, by continuing to address issues on a case-by- case basis, DOD risks that its solution to one issue could exacerbate another—for example, in addressing a security risk in isolation, DOD could inadvertently create further risks to data accessibility. According to F-35 program officials, a functional ALIS is key to the operational capability of the aircraft and the day-to-day ability to sustain the aircraft. Moreover, the department expects to significantly increase aircraft production within the next 5 years, so the number of aircraft that must be maintained and kept ready for flight will soon grow. By continuing to respond to issues on a case-by-case basis rather than in a holistic manner, there is no guarantee that DOD will address the highest risks, and as a result, DOD may encounter further schedule and development delays, including system upgrades, which could affect operations and potentially lead to cost increases.

DOD Has Estimated the Costs of ALIS, but Including Additional Analyses and Information Would Increase the Credibility and Accuracy of Its Estimate

DOD has estimated total ALIS costs to be approximately $16.7 billion over its 56-year life cycle. However, the estimate is not fully credible since DOD has not performed uncertainty and sensitivity analyses as part of its cost-estimating process. Moreover, while DOD has updated its estimate to be reflective of some program changes, it is also not fully accurate since DOD did not use historical cost data—both actual data from ALIS and data from comparable programs—when developing its ALIS estimate. Finally, other costs such as service customizations of ALIS may require additional future resources, and manual workarounds to the system currently require additional labor resources.

DOD Estimates that Total ALIS Costs Are about $16.7 Billion over a 56- Year Life Cycle

DOD estimates that total ALIS costs are about $16.7 billion—about $562 million to develop the system, about $1.1 billion to procure hardware and spare parts, and about $15.1 billion to sustain it in then-year dollars. DOD had expended approximately $505 million to develop ALIS as of December 2015, and the department estimates that continued development will cost an additional $57 million through 2017, which is when DOD expects ALIS 3.0 will be released for testing. In addition to the purchase cost for ALIS, DOD estimates that ALIS will cost about $15.1 billion to sustain over a 56-year life cycle. Program officials told us that ALIS development will be completed within the planned resources, but that the system will require follow-on modernization and that the program office is currently planning for those additional costs. Table 2 provides more detail on ALIS cost elements.

DOD’s Sustainment Cost Estimate for ALIS Is Not Fully Credible

We found that the program office’s estimate for ALIS sustainment costs minimally met the best practices for a credible cost estimate largely because it does not include uncertainty and sensitivity analyses (see app. II for more information on our assessment of the ALIS cost estimate). Every cost estimate contains a degree of uncertainty because of the many assumptions that must be made about the future. To mitigate this uncertainty, a variety of checks and analyses can be conducted to determine the credibility of the assumptions and the estimate as a whole. According to GAO’s Cost Estimating and Assessment Guide, cost estimates should include uncertainty analyses to determine the level of uncertainty associated with the estimate in order to be credible. A quantitative uncertainty analysis can provide a broad overall assessment of the risk in the cost estimate. In addition, credible cost estimates should include sensitivity analyses to examine how changes to individual assumptions and inputs affect the estimate as a whole.

Although ALIS is a multibillion dollar system, it is not a formally designated stand-alone weapons system program; therefore, DOD is not required to perform a separate estimate of the system’s projected costs. ALIS is one cost element within the overall F-35 program, and the program office estimates that the system will constitute less than 2 percent of the $891.1 billion total sustainment costs of the program. Cost estimators told us that despite its critical importance in operating and maintaining the entire F-35 fleet, since ALIS constitutes a small portion of the total estimate, it is not considered a major cost driver of the program. Therefore, cost estimators told us DOD’s cost-estimating guidance does not require them to perform uncertainty or sensitivity analyses for ALIS. As part of the planning process for F-35 sustainment, program officials have examined the role of F-35 information systems (which includes ALIS) in relation to five major value streams—maintenance, supply chain, training, management, and sustaining engineering. While program officials said this process helped map the connections and interdependence between program elements, it did not quantify in uncertainty or sensitivity analyses the potential effects of ALIS on sustainment costs specifically.

Additionally, DOD did not perform analyses to determine how further ALIS schedule delays or functionality issues could affect other F-35 costs. In lieu of the analyses, program officials stated that they assume that if ALIS does not perform as planned, aircraft could not be flown as frequently as intended, and this lower-than-expected utilization rate would therefore decrease sustainment costs. A 2013 DOD-commissioned study on reducing F-35 costs found areas of potential cost savings, but the implementation plan for this study also found that ALIS presents significant risk to the program. Particularly, this plan found that any functionality problems or schedule slippage with ALIS will have a significant impact on costs—with downstream additional costs due to performance and schedule delays potentially reaching up to $20-100 billion. The different conclusions drawn by the program office and this study suggest that DOD could better understand the effects that ALIS issues could have on overall program costs. Without performing uncertainty or sensitivity analyses, DOD will not understand how variabilities in ALIS-related assumptions could affect the estimate as a whole and the potential range of costs resulting from these variabilities.

DOD’s ALIS Cost Estimate Is Not Fully Accurate, and Upcoming Program Changes May Affect Future Estimates

We also assessed the ALIS estimate for accuracy and found that DOD partially met the standards for an accurate cost estimate. GAO’s Cost Estimating and Assessment Guide states a cost estimate should be based on historical data—both actual costs of the program and those of comparable programs—which can be used to challenge optimistic assumptions and bring more realism to a cost estimate. While we found that the DOD substantially met some best practices for an accurate cost estimate by properly adjusting for inflation and not including mathematical errors, the estimate uses contractor-provided data for material costs instead of actual ALIS costs or historical cost data from analogous programs that would make the estimate more accurate. Cost estimating officials said that they did not base their ALIS estimates on historical cost data because they believe that there are no programs analogous to ALIS. For example, there is a logistics system for the Air Force’s F-22 program—also a fifth-generation aircraft—but officials stated that is far less complex than ALIS and does not include all of ALIS’s applications and intended functions. However, multiple versions of ALIS have been fielded since 2010 and using historical data on known ALIS costs, as well as analogous data from the F-22 or other programs, would make the estimate more accurately representative of likely sustainment costs (see app. II for more information on our assessment).

GAO’s Cost Estimating and Assessment Guide also states that an estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—so that it is always reflecting current status. The program office updates its estimate annually and incorporates program changes and evolving assumptions in these updates, documenting the changes from year to year. However, the program office does not update all elements of the cost estimate. For example, technology refresh accounts for approximately 25 percent of ALIS sustainment costs and program officials were able to tell us the assumptions used to calculate these costs, however, they were unable to tell us, or identify within the estimating model, where these data came from or when their underlying assumptions were developed. Additionally, over the course of our review, program officials highlighted some recent or upcoming program changes, such as the need for additional infrastructure, that were not included in their last estimate.

The program office and services are exploring ways to decrease F-35 sustainment costs. For example, the services are taking different approaches to administering and maintaining ALIS, with the Marine Corps planning on using its own personnel, and the Air Force planning on using contractor support to administer the system, troubleshoot problems, and keep the system operating smoothly with software patches and other continuous improvements. Based on these service plans, the program office estimates that ALIS administration will cost more than $7 billion over the F-35 life cycle, in addition to about $1 billion for other contractor labor needs. Program officials said that these amounts may change based on upcoming sustainment decisions, including a potential way to decrease administrative costs by establishing regional centers that could provide this support to a number of F-35 sites rather than having contracted administrators at each site.

Other program changes have the potential to increase ALIS costs and future estimates. Although DOD has included estimated costs of ALIS technology refresh and software licensing, program officials have stated that the current estimate does not sufficiently capture the full costs of follow-on modernization that will be required when upgrades or new versions of the system’s commercial off-the-shelf software are released. Because ALIS comprises multiple applications and interfaces with service networks and legacy information systems, engineers will have to reintegrate ALIS with other systems as well as the applications within ALIS in step with continuous upgrades and software improvements. Although DOD expects ALIS 3.0 to be the fully capable version that meets program requirements, officials at the program office and from all three services stressed that ALIS will not be a static system and that improvements and upgrades will not only be expected, but required, to keep ahead of technology obsolescence and evolve with emerging threat environments. Program officials stated that there is a need to build these follow-on modernization costs into the budgeting and cost-estimating processes.

Program officials also stated that they may procure up to two more ALOUs for back-up and necessary redundancy and that, in addition to addressing the current risk of the single ALOU becoming a single point of failure, these additional ALOUs may facilitate greater government operation of ALIS and increase the potential for greater competition of future sustainment contracts. The program office bases much of its estimate on inputs from the services, such as their expected personnel needs and how they plan to operate and sustain their F-35 squadrons. The program plans to field one SOU for each F-35 squadron, and another SOU at each forward operating location. Since deploying squadrons do not normally transit to the forward location together as one unit but rather in a more staggered fashion, having an SOU at both the squadron’s home base and another prepositioned at the forward deployed location would avoid potential problems of having a split squadron sustained with just one SOU. Having additional SOUs positioned at forward deployed locations would likely increase procurement costs and the downstream costs of maintaining and replacing them.

While the program office has been incorporating some program changes and adjusting its cost estimating assumptions as the F-35 program grows and evolves, it is important to continue this effort in its annual estimate updates to reflect current or planned program changes such as those described above. Additionally, as the program gains more experience operating ALIS, using historical data—especially actual program costs as they become available—as the basis for its estimates can result in a more accurate and realistic picture of ALIS costs. Unless DOD’s estimate is based on an assessment of the most likely costs, the estimate may not be representative of how much it will cost to sustain ALIS and may inhibit informed decision making.

Service Customizations and Workarounds Used to Overcome ALIS Limitations Could Require Additional Resources

There are other ALIS-related costs that are not included in the estimate but that may place additional financial or manpower burdens on the services as ALIS continues to be concurrently developed and fielded. Services are responsible for bearing the costs of engineering any service- specific ALIS customizations. According to DOD officials, ALIS engineering changes must be agreed-upon by all services and partners in order to become new requirements. The program office then communicates these requested changes to the contractor software engineers developing ALIS. However, as ALIS users become more familiar with the system and its limitations, the services may request additional changes to the design of ALIS to meet their specific airworthiness and reporting requirements, and incur the additional costs to meet these needs.

Some service-specific reporting requirements are being met through the use of time-consuming workarounds employed by ALIS users to compensate for current system limitations. According to program officials, some service-specific reporting requirements are not addressed by current ALIS functionality. For example, maintainers at several sites said that they must request maintenance data entered and housed within ALIS from the contractor because they cannot access this information independently. They then use these raw data to generate reports on aircraft status using software programs outside of ALIS because the system does not have the capability to extract and process the data in the form that the services require. Program officials told us that they expect some of these workarounds to become unnecessary after upcoming system improvements. Other ALIS functionality issues that create the need for workarounds do not have scheduled solutions, so both program and service headquarters officials are in the process of examining the use of these workarounds to determine whether they are truly necessary and will require an ALIS design change or whether service expectations should be better managed. Program officials told us that this workaround is not a problem with ALIS itself, but more an issue of the services requiring reports that the system was not designed to provide. Users at all the sites we visited told us that ALIS should have the functionality to create reports that they need.

Program officials said that there is not an extra financial cost associated with these workarounds since they are performed by service personnel, but rather there is an opportunity cost to performing them—the additional time personnel spend on manually analyzing data and generating reports rather than performing their primary job duties. Neither the program office nor the services track the use of ALIS workarounds across all F-35 squadrons, but squadron leadership at Eglin Air Force Base provided examples of workarounds performed at their site including the estimated amount of time personnel must spend to overcome ALIS shortcomings. They estimate that personnel at that location alone spend approximately 150 man-hours per week on these workarounds—nearly the equivalent of having four full-time personnel dedicated to manually creating reports because, according to officials, ALIS currently cannot. Eglin Air Force Base personnel have created and updated 14 manual products mainly because of ALIS report limitations and because ALIS does not track information needed by senior leaders for unit-level analysis and subsequent maintenance and logistics decisions. They added that without these manual products, analysis, scheduling, and maintenance operations would suffer degradation and that mission accomplishment could be jeopardized.

The use of workarounds varied across the F-35 sites we visited, and officials said that Eglin Air Force Base’s examples are not necessarily representative of other sites’ workarounds. They provide a sense of scale, however, for the labor burden placed on personnel because of current system immaturity and functionality issues. Personnel at F-35 sites we visited told us that the extra time they spend on ALIS workarounds detracts from the time they have to maintain proficiency in their specialties, prevents them from coaching and training their subordinates, and decreases the amount of time they have to perform collateral duties. They added that ALIS is not yet fully autonomic and will require significant additional system improvements for it to perform to their expectations.

The use of these ALIS workarounds highlights the current immature state of the system and underscores the need for DOD to prioritize, and address key ALIS risks as discussed earlier in this report. As the F-35 program increases production over the next several years, sustained attention to addressing these issues and improving estimates of ALIS costs can help decision makers better direct resources and ensure that ALIS meets the needs of its users.

DOD Does Not Have a Program-Wide Training Plan for ALIS Users but Has Taken Some Initial Steps to Address Training Shortfalls

DOD does not have a program-wide training plan for ALIS, but has taken initial steps to address training shortfalls. According to ALIS users at all five F-35 operational and training sites where we conducted our focus groups, training for ALIS is ineffective, and lacks a standardized, common curriculum for teaching users how to operate ALIS. Basic ALIS training courses are made available to all ALIS users at the F-35 Academic Training Center at Eglin Air Force Base. The training, according to ALIS users across all five sites, consists of a series of PowerPoint slides that are geared toward illustrating the conceptual nature of the system— showing how all the applications within ALIS are supposed to work— rather than how to actually operate the system as it currently exists. Furthermore, several ALIS users described the training as ineffective because it does not teach users how to explicitly operate specific ALIS applications. For example, the slideshow-based classroom training, which is optional for ALIS users, is not customized for the different users of ALIS. Pilots, maintainers, supply personnel, data analysts, and other F-35 specialists are required to use certain ALIS applications, and often in different ways based on their job requirements. Maintainers at two of the five sites said that the classroom training reputation has become so bad that many new maintainers choose to skip the courses and proceed directly to on-the-job training when they begin working on the F-35.

From the outset of the original ALIS release in 2010, DOD has relied on the contractor to manage all ALIS training. ALIS training is currently heavily dependent on learning on the job, which consists of users learning how to operate ALIS from colleagues and through trial and error. According to DOD officials, this is not an uncommon practice for new systems within the department; however, those officials agree that training typically begins with a common classroom curriculum, and is supplemented with on-the-job-training. Almost every ALIS user in our focus groups noted that they do not learn how to operate any ALIS applications until on-the-job-training begins on the flight line. Specifically, the classroom training does not afford an opportunity to practice on “sandbox” or “ghost” systems that would simulate how ALIS is used on a daily basis by users. Instead, the practice comes in a live operational environment, where basic ALIS functions and practices are taught by supervisors or other users within the respective squadron. According to ALIS users at all five sites, this practice has led to users learning inconsistent methods and shortcuts for maneuvering through the different ALIS applications. In most cases, these practices differ between squadrons and services; however, there were also cases that some maintainers highlighted where users were learning different practices within the same squadron. ALIS users reported that they have created different workarounds to overcome ALIS functionality issues, but have also cited instances of not using ALIS as intended because the system is unwieldy and time-consuming. ALIS users said they are learning to operate the system in different ways and then perpetuating these methods, creating a situation where ALIS may not be operated in the most effective, efficient, or up-to-date way across all F-35 sites. Furthermore, they are learning to use the system in a live operational environment, running the risk of making errors that could ultimately affect aircraft availability.

Program officials acknowledged the training shortfalls identified in the focus-group sessions. In response to these shortfalls, the program office has taken some initial steps to address some of the issues. Specifically, the program office, in concert with the contractor, has developed Mobile Training Teams to offer a way to train ALIS users outside of the F-35 Academic Training Center and at their specific F-35 sites. These teams are deployed to F-35 operational and testing sites to help keep ALIS users up-to-date with ALIS software version releases, which, according to ALIS users at all sites we visited, had been a significant problem with ALIS training. Specifically, according to ALIS users at all these sites, ALIS trainers and administrators are rarely up-to-date on the latest ALIS releases and functionality changes; therefore, it has led to inconsistencies in teachings and practices at the squadron level. Mobile Training Teams offer ALIS version-specific training at each site based on a sequencing schedule developed by the program office.

In addition, the program office has rolled out an ALIS Training Evaluation to determine the current state of ALIS training across all F-35 sites. The end goal of this evaluative process is to identify underlying training deficiencies, and to develop corrective courses of action to mitigate these deficiencies. The process will include a series of site visits to include course audits, interviews, curricula inspections, and stakeholder surveys as applicable for root-cause analysis. As of January 2016, a team within the program office had just begun this process; therefore, they could not provide any details beyond the effort’s scope, methodology, and associated time frames.

According to best practices of information-technology training, effective training of users is essential to the workforce supporting an information- technology system. The practices suggest that entities develop Strategic Learning Plans, or Overarching Training Plans, to help align training programs with priorities. Furthermore, as part of this process, the practices state that it is important that the training design and delivery process ensures learning occurs during the training and also ensures that the user applies the training on the job. Additional guidance on strategic training in the federal government states that training plans can aid in the performance of government programs. Specifically, a training plan can present a business case for proposed training and development investments, including the identified problem or opportunity, the concept for an improved situation or condition, and linkages with strategic objectives.

According to DOD officials, ALIS training has been a difficult process to manage because of the dynamic nature in which ALIS has been developed and upgraded since its initial release. Because new versions of ALIS have been regularly released in a staggered manner across F-35 sites, they said it has been difficult to sufficiently train all ALIS users on the most up-to-date versions and teach consistent practices. However, with only one major version upgrade remaining prior to version 3.0 (the final major software release), issues related to constantly changing the system should decrease. A standardized, program-wide training plan could remove the emphasis from on-the-job training and provide a comprehensive, standardized training curriculum across the program. Without a program-wide training plan that assures that consistent learning occurs in the classroom, and is then applied by users on the job, the program runs the risk of continuing to allow users to learn irregular or incorrect practices through a training culture driven by on-the-job training, which could impact aircraft availability and safety.

Conclusions

The F-35 program is DOD’s largest and most costly acquisition program to date. According to senior defense officials, not only does the F-35 program represent the future of tactical fighter aircraft in our military, but it is also vital to the security of our nation moving forward. With the Marine Corps already having declared the F-35 both deployable and combat- ready, the Air Force and Navy set to do the same within the next 2 years, and the full-rate production of the aircraft set for 2019, including the ramp up of sustainment activities, it is imperative that DOD address major risks associated with its central logistics system—ALIS. The program office has taken steps to identify risks associated with ALIS, including all of those identified by participants in our focus groups—and has begun, on a case- by-case basis, to address some of these risks; however, without a plan to prioritize risks and address them in a systematic and holistic manner, DOD runs the risk of having an ALIS that is not fully functional as it approaches key program milestones. Without a fully functional ALIS, DOD could face operational and schedule risks and potential cost increases to a program that is already the most expensive in DOD’s history.

Although DOD has estimated the costs of ALIS, additional information would increase the accuracy and credibility of the estimate. While ALIS is projected to constitute less than 2 percent of the $891.1 billion total sustainment costs of the program, the financial impact that nonfunctional aspects of ALIS may have on the overall operations and sustainment of the aircraft could be significant. Until DOD does more analyses to determine the impact ALIS has (and the impact a nonfunctional ALIS could have) on overall sustainment costs, DOD will not know how much the costs of ALIS and overall sustainment could fluctuate. Furthermore, while the program office has been incorporating some program changes and adjusting its cost-estimating assumptions for ALIS, it is important for the program office to improve the reliability of its estimate by using historical data. It is also important for DOD to incorporate program changes that will likely affect ALIS costs in future estimates, such as decisions to enhance ALIS infrastructure or decrease planned numbers of administrative personnel with regional support centers.

Finally, although DOD has recognized that ALIS training needs improvement and has made some temporary fixes to address the current shortcomings, DOD has yet to develop a program-wide training plan that would take the focus off an almost explicit on-the-job-training approach, and provide greater consistency among ALIS users. Considering the importance ALIS has to the operations and sustainment of the aircraft, and that DOD plans to purchase and operate nearly 500 more F-35s in the next 5 years, it will be important that ALIS users be on the same page with regard to operating the system. Further training inconsistencies could lead to impacts on aircraft availability and safety.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct the F-35 Program Executive Officer to take the following four actions:

To ensure that risks associated with ALIS are addressed expediently and holistically, develop a plan that would prioritize and address ALIS issues, prior to the start of full-rate production for the program.

To improve the reliability of its cost estimates, conduct uncertainty and sensitivity analyses consistent with cost-estimating best practices identified in GAO’s Cost Estimating and Assessment Guide.

To improve the reliability of its cost estimates, ensure that future estimates of ALIS costs use historical data as available and reflect significant program changes consistent with cost-estimating best practices identified in GAO’s Cost Estimating and Assessment Guide.

To ensure that ALIS training issues are fully addressed, develop a standardized, program-wide plan for ALIS training through the life cycle of the program.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with two recommendations and partially concurred with two recommendations. DOD’s comments are summarized below and reprinted in appendix III. DOD also provided technical comments, which we have incorporated into our report where appropriate.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to develop a plan that would prioritize and address ALIS issues, prior to the start of full-rate production for the program. DOD stated that the F-35 program office began developing an ALIS Technical Roadmap in early 2016. The department added that at its completion later in 2016, this roadmap will be the foundation of a plan to identify, document, and prioritize ALIS risks; address them holistically; and inform budget priorities, as the program transitions from development into sustainment and follow-on modernization. Additionally, to mitigate the risk of single-point failures in the infrastructure, the program office contracted to acquire backup ALIS hardware in 2015; the backup hardware will be operational by early 2017. We state in our report that the department was aware of the risks to ALIS and believe that if DOD develops a plan to identify, document, and prioritize ALIS risks, address them holistically, and inform budget priorities prior to full-rate production, this action should address our recommendation. We further state in our report that DOD was in the early stages of acquiring backup ALIS hardware to mitigate the risk of single- point failures in the infrastructure, but had not yet allotted funding. We believe this backup system will be critical as the program approaches full- rate production.

DOD partially concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to conduct uncertainty and sensitivity analyses consistent with cost-estimating best practices identified in GAO’s Cost Estimating and Assessment Guide. DOD stated that the department considers the sensitivity analyses that the F-35 program office performs to be a form of uncertainty analysis, as described in DOD’s Cost Assessment and Program Evaluation Operating & Support Cost Estimating Guide; however, the DOD cost estimating guidance does not require DOD to conduct a sensitivity or uncertainty analysis on ALIS since DOD does not consider ALIS a major cost driver of the F-35 program. As our report states, according to GAO’s Cost Estimating and Assessment Guide, cost estimates should include uncertainty analyses to determine the level of uncertainty associated with the estimate in order to be credible. A quantitative uncertainty analysis can provide a broad overall assessment of the risk in the cost estimate. A sensitivity analysis can examine how changes to individual assumptions and inputs affect the estimate as a whole. Although the F-35 program office may conduct analyses consistent with DOD cost estimating guidance, it has not conducted an uncertainty or sensitivity analysis specifically for ALIS. Although ALIS is projected to constitute less than 2 percent of the $891.1 billion total sustainment costs of the program, the financial impact that nonfunctional aspects of ALIS may have on the overall operations and sustainment of the aircraft could be significant. For example, a 2013 DOD-commissioned plan found that any functionality problems or schedule slippage with ALIS will have a significant impact on costs—with downstream additional costs due to performance and schedule delays potentially reaching up to $20-100 billion. We continue to believe that without completing uncertainty and sensitivity analyses to determine the effect ALIS has (and the impact a nonfunctional ALIS could have) on overall sustainment costs, DOD will not know how much the costs of ALIS and overall sustainment could fluctuate.

DOD partially concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to ensure that future estimates of ALIS costs use historical data as available and reflect significant program changes consistent with cost-estimating best practices identified in GAO’s Cost Estimating and Assessment Guide. DOD stated that the department will ensure that the future F-35 program ALIS cost estimates continue to use the latest available historical cost data as appropriate and reflect the latest approved technical baseline when the program office incorporates these into the program of record, according to DOD’s Cost Assessment and Program Evaluation Operating and Support Cost Estimating Guide; however, we found that DOD was not using all available historical data. As our report states, according to GAO’s Cost Estimating and Assessment Guide, a cost estimate should be based on historical data—both actual costs of the program and those of comparable programs—which can be used to challenge optimistic assumptions and bring more realism to a cost estimate. While we found that DOD substantially met some best practices for an accurate cost estimate by properly adjusting for inflation and not including mathematical errors, the estimate uses contractor-provided data for material costs instead of actual ALIS costs or historical cost data from analogous programs that would make the estimate more accurate. We continue to believe that it is important for the program to improve the reliability of its ALIS estimate by using historical data to the greatest extent possible. It is also important for DOD to incorporate program changes that will likely affect ALIS costs in future estimates, such as decisions to enhance ALIS infrastructure or decrease planned numbers of administrative personnel for regional support centers.

DOD concurred with the recommendation that the Secretary of Defense direct the F-35 Program Executive Officer to develop a standardized, program-wide plan for ALIS training through the life cycle of the program. DOD stated that, to address immediate issues, the F-35 program office deployed Mobile Training Teams to assist ALIS users at their home base locations, and to address longer-term issues, the program office began a comprehensive evaluation of ALIS training in 2015. According to DOD, the completion of this evaluation in 2016 will inform development of a plan to address long-term ALIS training issues. We agree and state in our report that the F-35 program has taken some positive steps to address short-term training shortfalls by deploying Mobile Training Teams as a way to train ALIS users outside of the F-35 Academic Training Center. We also report that the program has recently begun a comprehensive evaluation of ALIS training to determine the current state of ALIS training across all F-35 sites. If the F-35 program leverages this comprehensive evaluation, when it is completed, to develop a program-wide plan for ALIS training through the life cycle of the program, this action should address our recommendation.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff has any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To determine the extent to which the Department of Defense (DOD) has a plan to ensure the Autonomic Logistics Information System (ALIS) is fully functional as the key F-35 program milestones approach, we reviewed documentation of program plans with relevant sustainment elements including the F-35 Global Sustainment Plan, the Weapon System Planning Document, the F-35 Autonomic Logistics Global Sustainment Concept of Operations, and the F-35 Operational Requirements Document. We also selected and conducted site visits at a nongeneralizable sample of five F-35 operational and training sites:

Eglin Air Force Base, Florida

Luke Air Force Base, Arizona

Edwards Air Force Base, California

Nellis Air Force Base, Nevada

Marine Corps Air Base Yuma, Arizona We selected these sites in consultation with service officials to ensure we obtained perspectives across all three services and at both operational and testing sites. During these visits, we convened 17 non-generalizable focus-group sessions with a range of ALIS users from all three services to obtain information on the operability and deployability of ALIS, and how any ALIS issues may pose risks for F-35 operations and sustainment. Specifically, we convened groups of maintainers, pilots, system administrators, and trainers. We also held focus groups with contractor personnel responsible for training and administering ALIS at these sites. There were approximately a total of 120 participants in these focus groups. Table 3 includes a breakdown of the focus groups we held at the various locations.

We worked with our methodologist to develop a focus group script that included questions across four main categories: Training, ALIS Positives, ALIS Negatives, and Risks, that was used at all five site visits. For consistency, our methodologist facilitated all focus group sessions. To analyze the focus-group responses, we conducted content analyses, developing categories and sub-categories and coding comments from each focus group to these categories. After each comment had been coded by an analyst, another analyst independently reviewed each code and either agreed or disagreed with the coding decision. Where there was disagreement in the coding decision, the two analysts discussed it and came to a resolution. Based on the content analysis, we described the overarching ALIS benefits and risks obtained from ALIS users about the system and any risks it poses to the F-35 program as it approaches key milestones. After obtaining information on DOD’s current approach to addressing the functionality issues that ALIS users identified, we evaluated information we obtained for consistency with best practices from GAO’s Schedule Assessment Guide and DOD’s System Engineering Guide for System of Systems that provide guidance and best practices on how, prior to meeting key milestones, a plan to address specific risks that may be associated with major weapon acquisitions should be developed. We also interviewed key DOD and contractor officials to collect information about building and testing ALIS, the capabilities of ALIS, metrics collected on ALIS’s development and performance, software upgrades to ALIS, and how the F-35 program office is addressing ALIS functionality issues.

To determine the extent to which DOD has credibly and accurately estimated ALIS costs, we evaluated the reliability of DOD’s estimate of ALIS costs contained in the F-35 program office’s 2014 estimate of F-35 operating and support (O&S) costs, the most up-to-date estimate completed at the time of our review. The program office completed the 2014 cost estimate in the spring of 2015 and plans to release its 2015 updated estimate in spring of 2016. The Office of the Director for Cost Assessment and Program Evaluation (CAPE) also performed an F-35 O&S cost estimate in 2013, but has not updated it. Both CAPE and Joint Program Office officials told us they used the same cost inputs, methodology, and ground rules and assumptions when estimating ALIS costs in their respective estimates. We used characteristics contained in GAO’s Cost Estimating and Assessment Guide to assess the reliability of DOD’s estimate of ALIS costs. According to the guide, there are four general characteristics of sound cost estimating: being well-documented, comprehensive, credible, and accurate. For the purposes of this review, we conducted a limited assessment and used two of the four general characteristics of sound cost estimating included in this guide: being credible and accurate. We chose these characteristics because ALIS costs represent only one element of the total F-35 cost estimate—less than 2 percent of projected F-35 sustainment costs—and therefore we determined that it would not be appropriate to assess whether the estimate was comprehensive or well-documented. To determine whether the credible and accurate characteristics were met, we reviewed documentation used to generate the program office’s estimate, including data sources, assumptions, and cost models, and we interviewed cost- estimating officials from the program office and CAPE. We also interviewed other officials from the program office and service headquarters to discuss the cost effects of ALIS schedule delays and development issues. Results of our assessment of the estimate’s credibility and accuracy, along with descriptions of these characteristics and their associated best practices, are detailed in appendix II of this report. We found the data to not be fully reliable, which we discuss in further detail in the report and in appendix II.

Finally, to determine the extent to which DOD has developed a plan to manage ALIS training for users, we reviewed key documentation related to ALIS and F-35 training, and used information from our focus-group sessions across various types of ALIS users from all three services to obtain information on the current state of ALIS training. We also interviewed key DOD and contractor officials. We evaluated all of the information we received using GAO-developed and industry best practices for information-technology training, and DOD’s Policies and Procedures for Acquisition of Information Technology.

To address all of our objectives, we collected and analyzed information and interviewed officials from the following Department of Defense (DOD) offices:

Office of the Under Secretary of Defense (Acquisitions, Technology and Logistics)

Office of the Director for Cost Assessment and Program Evaluation (CAPE)

Office of the Director for Operational Test and Evaluation (DOT&E)

Department of the Air Force

Department of the Navy

Headquarters Marine Corps

F-35 Joint Program Office We also collected and analyzed information and interviewed officials from Lockheed Martin in Fort Worth, Texas, and Orlando, Florida.

We conducted this performance audit from April 2015 to April 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides and reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Scoring of the F-35 Program Office’s 2014 Autonomic Logistics Information System (ALIS) Operating and Support (O&S) Cost Estimate

We completed an assessment of the ALIS costs in the F-35 program office overall F-35 O&S cost estimate on the basis of two characteristics—being credible and accurate—and their associated best practices derived from the GAO Cost Estimating and Assessment Guide. After reviewing documentation that the program office submitted for its 2014 F-35 ALIS O&S cost estimate, conducting interviews with program office cost-estimating officials, and reviewing relevant sources, we determined that these cost estimates are not fully reliable. While we found that the program office estimate for ALIS is partially accurate, the estimate is minimally credible. These evaluations are shown in table 4. We determined the overall assessment rating by assigning each individual best practice rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the average of the individual best practice assessment ratings to determine the overall rating for each of the two characteristics. The resulting average becomes the Overall Assessment as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered reliable if the overall assessment ratings for each of the two characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition the contact name above, the following staff members made key contributions to this report: Alissa Czyz, Assistant Director; Steven Banovac; Jeffrey Hubbard; Jason Lee; Jennie Leotta; Terry Richardson; Amie Lesser; Alyssa Weir; and Delia Zee.

Related GAO Products

F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016.

F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015.

F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014.

F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26 2014.

F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014.

F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013.

F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013.

F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013.

Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012 Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-12-400SP. Washington, D.C.: March 29, 2012.

Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012.

Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.

Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.

Tactical Aircraft: Air Force Fighter Force Structure Reports Generally Addressed Congressional Mandates, but Reflected Dated Plans and Guidance, and Limited Analyses. GAO-11-323R. Washington, D.C.: February 24, 2011.

Defense Management: DOD Needs to Monitor and Assess Corrective Actions Resulting from Its Corrosion Study of the F-35 Joint Strike Fighter. GAO-11-171R. Washington D.C.: December 16, 2010.

Joint Strike Fighter: Assessment of DOD’s Funding Projection for the F136 Alternate Engine. GAO-10-1020R. Washington, D.C.: September 15, 2010.

Tactical Aircraft: DOD’s Ability to Meet Future Requirements Is Uncertain, with Key Analyses Needed to Inform Upcoming Investment Decisions. GAO-10-789. Washington, D.C.: July 29, 2010.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-10-388SP. Washington, D.C.: March 30, 2010.

Joint Strike Fighter: Significant Challenges and Decisions Ahead. GAO-10-478T. Washington, D.C.: March 24, 2010.

Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting Warfighter Requirements on Time. GAO-10-382. Washington, D.C.: March 19, 2010.

Joint Strike Fighter: Significant Challenges Remain as DOD Restructures Program. GAO-10-520T. Washington, D.C.: March 11, 2010.

Joint Strike Fighter: Strong Risk Management Essential as Program Enters Most Challenging Phase. GAO-09-711T. Washington, D.C.: May 20, 2009.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-09-326SP. Washington, D.C.: March 30, 2009.

Joint Strike Fighter: Accelerating Procurement before Completing Development Increases the Government’s Financial Risk. GAO-09-303. Washington D.C.: March 12, 2009.

Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. GAO-08-782T. Washington, D.C.: June 3, 2008.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-08-467SP. Washington, D.C.: March 31, 2008.

Joint Strike Fighter: Impact of Recent Decisions on Program Risks. GAO-08-569T. Washington, D.C.: March 11, 2008.

Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks. GAO-08-388. Washington, D.C.: March 11, 2008.

Tactical Aircraft: DOD Needs a Joint and Integrated Investment Strategy. GAO-07-415. Washington, D.C.: April 2, 2007.

Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter Engine Program. GAO-07-656T. Washington, D.C.: March 22, 2007.

Joint Strike Fighter: Progress Made and Challenges Remain. GAO-07-360. Washington, D.C.: March 15, 2007.

Tactical Aircraft: DOD’s Cancellation of the Joint Strike Fighter Alternate Engine Program Was Not Based on a Comprehensive Analysis. GAO-06-717R. Washington, D.C.: May 22, 2006.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

Defense Acquisitions: Actions Needed to Get Better Results on Weapons Systems Investments. GAO-06-585T. Washington, D.C.: April 5, 2006.

Tactical Aircraft: Recapitalization Goals Are Not Supported by Knowledge-Based F-22A and JSF Business Cases. GAO-06-487T. Washington, D.C.: March 16, 2006.

Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance. GAO-06-356. Washington, D.C.: March 15, 2006.

Joint Strike Fighter: Management of the Technology Transfer Process. GAO-06-364. Washington, D.C.: March 14, 2006.

Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications for Tactical Aircraft Modernization. GAO-05-519T. Washington, D.C: April 6, 2005.

Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy. GAO-05-271. Washington, D.C.: March 15, 2005.