Air Force Training: Further Analysis and Planning Needed to Improve Effectiveness
Released: 2016-09-19T00:00:00

Letter

Background

The Secretary of the Air Force is responsible for recruiting, organizing, training, and equipping forces in order to meet the current and future operational requirements of DOD’s combatant commands. The Air Force is organized into 10 major commands that are assigned a major part of the Air Force mission. Numbered Air Forces fall directly under the major commands and are focused on ensuring the readiness of assigned forces. For example, Pacific Air Forces contains three numbered air forces that are located in South Korea, Japan, and Alaska. They provide operational leadership and supervision and are assigned subordinate units, such as wings, groups, and squadrons. The component numbered Air Force is a specialized category of numbered Air Force that is structured to perform an operational and warfighting mission in support of a combatant command. Air Force units, such as an operational flying squadron, are designed to provide a specific set of mission capabilities to meet anticipated operational requirements. See figure 1 for general characteristics of the combat aircraft platforms included in our review. Each combat unit has a Designed Operational Capability statement that it uses for reporting readiness data and includes, among other things, the full range of core missions that the unit can be called on to provide. Air Combat Command has responsibility for developing and managing aircrew training requirements to meet the full range of core missions.

Throughout their careers, Air Force personnel designated as aircrews participate in several different types of training. When they first enter the Air Force, they receive basic military training. After basic military training, aircrews receive additional training, including undergraduate training by the Air Force’s Air Education and Training Command. Next, aircrews receive initial qualification training that is focused on their career specialties and is provided by the Air Education and Training Command for the F-16, F-15C, and F-35 platforms and by Air Combat Command for the A-10, F-15E, and F-22 platforms. After aircrews complete initial qualification training, they are assigned to a unit, where they complete mission qualification training—additional training on the skills specific to the unit’s capabilities. Air Force Instruction (AFI) 11-202 is the foundational aircrew training instruction that guides training programs for all Air Force flying training. There is an AFI 11-2, Volume 1 for each aircraft platform that specifies the training requirements for aircrews who are assigned to units that operate that platform. For example, F-22 units would use AFI 11-2F-22A, Volume 1. In addition to AFI 11-202, tasking memorandums for the Ready Aircrew Program specify annual continuation training for personnel assigned to combat units. Specifically, Ready Aircrew Program tasking memorandums establish the minimum number of live training events (“sorties”) and simulator missions (or virtual training) that aircrews must complete during the annual training cycle to be considered combat mission ready or basic mission capable for the assigned aircraft. Finally, aircrews may complete upgrade training to qualify for positions such as flight leader, instructor pilot, or mission commander. “Combat mission ready” is a squadron commander certification based on the extent to which aircrews or units have completed and sustained required training. Combat mission ready status is maintained by completing the annual continuation training requirements as prescribed by the unit’s AFI 11-2, Volume 1, and the Ready Aircrew Program tasking memorandums. These requirements include completing a specific quantity of relatively advanced individual and unit training events within a given time period. For example, experienced F-22 aircrews must perform 11 defensive counter air training sorties in a 12-month period. In addition, aircrews must achieve and maintain “currency” by completing certain required training events deemed critical to safety of flight, such as landings, low altitude flight, and formation take-offs. For example, experienced F-22 aircrews are to land the aircraft every 45 days or else they would lose landing currency. Currency events are typically achieved by completing the annual training requirements.

Most Air Force combat units report the status of their unit training through the Air Combat Command’s Ready Aircrew Program. The Ready Aircrew Program establishes the minimum number of sorties and simulator missions that aircrews must accomplish to sustain combat mission readiness. These sorties and simulator missions are aligned with the unit’s primary missions, for which the units must maintain “proficiency,” and secondary missions, for which they must maintain “familiarity.” For example, the fiscal year 2016 tasking memorandum for the F-15E aircraft specifies that experienced aircrews are required to complete a minimum of 96 annual live sorties (8 per month) to be considered combat mission ready, of which 49 are flown to fulfill primary mission requirements. Similarly, inexperienced aircrews are to complete 108 annual sorties (9 per month), of which 54 are flown to fulfill primary mission requirements. Both experienced and inexperienced aircrews are required to complete 36 annual simulator missions (3 simulator missions per month) to be considered combat mission ready. Units are required to submit an annual end-of-cycle training report covering the 12-month training cycle. These reports include information on the percentage of training accomplished by the unit against the training requirements for its primary and secondary missions, as well as any factors that limit the unit’s ability to meet its training requirements, such as the availability of personnel or aircraft.

Virtual training uses aircraft simulators that replicate the live environment for various missions and events. Aircraft simulators can be standalone or linked to information networks that bring together geographically- separated simulators in order to conduct training. Training missions that connect geographically-separated simulators are referred to as distributed mission operations. Distributed mission operations utilize the integration of virtual (e.g., a person training in a simulator) and constructive (e.g., computer-generated) elements to train aircrews. To date, Air Force virtual training efforts have focused mainly on the integration of virtual and constructive aspects of distributed training. However, the Air Force’s goal is to integrate live training with distributed mission operations, which is referred to as Live, Virtual, and Constructive (LVC) operational training.

The Air Force’s Process for Developing Combat Aircrew Annual Training Requirements for the Full Range of Core Missions Is Based on Dated Assumptions

The Air Force establishes combat aircrew training requirements for the full range of core missions based on an annual process, but these requirements may not reflect current and emerging training needs, because the Air Force has not comprehensively reassessed the assumptions underlying them. The Air Force process to develop annual training requirements for the aircrews of each type of combat aircraft includes input from the component numbered Air Forces and concludes with an annual training review board that brings together stakeholders. However, these requirements may not reflect current and emerging training needs, because the Air Force has not comprehensively reassessed the assumptions underlying the total number of annual live-fly training sorties by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training since 2012.

Air Combat Command Has a Process to Develop Annual Training Requirements for the Full Range of Core Missions

The Air Force has established a process to develop annual training requirements for the aircrews of each type of combat aircraft for the full range of core missions. This process begins with identifying the capability needs of the component numbered Air Forces. Every two years the Vice Commander of Air Combat Command sends a letter to the component numbered Air Forces requesting that they review their primary and secondary mission lists and identify their priorities for Air Force units by mission and type of combat aircraft. The component numbered Air Forces provide their responses to Air Combat Command, describing their expectations and priorities.

Annually, Air Combat Command takes the expectations and priorities identified by the component numbered Air Forces and develops the draft Ready Aircrew Program tasking memorandums that set the annual training requirements for each aircraft. Inputs into development of the draft tasking memorandums include subject-matter experts’ assessments of available training resources and the mid-year and end-of-year training and readiness reviews. The draft Ready Aircrew Program tasking memorandums are then released to the fighter wings to obtain feedback from the operations group commanders on their units’ ability to execute the training requirements. Air Combat Command officials consolidate group commander responses and prepare the agenda for the annual Realistic Training Review Board. The review board consists of senior officers who meet over three days during the summer to review the draft Ready Aircrew Program tasking memorandums and make their final recommendations to Air Combat Command. The final adjustments are made by Air Combat Command, and the Ready Aircrew Program tasking memorandums are published before the start of the next fiscal year. Figure 2 summarizes Air Combat Command’s process for developing these annual training requirements.

The annual training requirements are used to estimate the Air Force’s budget request for the flying-hour program, which funds live training. These hours are determined using the Air Force Single Flying Hour Model, which provides the methodology and processes that the major commands use to build their flying-hour programs. The model calculates the flying hours needed based on inputs of the force structure (number of aircraft and aircrew), aircrew data (types and number of aircrew that require training), and live-fly requirements (established in the Ready Aircrew Program tasking memorandums for combat air forces). Headquarters Air Force has overall program management responsibility for the flying-hour program, and major commands are responsible for their requirements and for reporting on how those hours are used. Air Combat Command performs this function for the combat air forces. Once flying hours are allocated to units, the wing commanders manage those hours and report monthly to Headquarters Air Force on the number of hours actually flown.

Annual Training Requirements Are Based on Dated Assumptions and May Not Reflect Current and Emerging Training Needs

The Air Force strategy, America’s Air Force: A Call to the Future, discusses the need to ensure a full-spectrum-capable force that can successfully achieve missions to address a broad range of current and emerging threats. Further, the document states that the design of training should move to achieve sufficient readiness across all mission sets while utilizing virtual training to deliver robust and realistic training against existing and emerging threats. On an annual basis, Air Combat Command makes updates to its training plans based on the process discussed above. These updates can include changes to the aircraft’s primary and secondary missions, the allocation of sorties among missions, and the types of required training events, among other changes. However, based on our review of annual training requirements for combat aircrews from 2012 through 2016 and discussions with Air Force officials, we found that Air Combat Command has used the same underlying assumptions to establish its annual training requirements. Specifically, the total annual live-fly training sorties by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training remained the same during this time period and may not reflect current and emerging training needs, as discussed below.

Since 2012, Air Combat Command has set the same requirement for the minimum number of live-fly sorties across all combat aircraft platforms, regardless of the number of core missions assigned to each platform. For fiscal year 2016, for example, the annual requirement for live training sorties for an F-15C, which has two primary missions and no secondary missions, is the same as the annual requirement for an F-35, which has three primary missions and three secondary missions. Officials from Air Combat Command stated that the live-fly sortie requirements are based in part on a 2010 Navy analysis of the number of sorties needed to perform safe flying operations. However, two of four wing commanders we spoke with noted that differences in the number of core missions should be considered when establishing minimum requirements for live training sorties by platform, because of variations in the level of complexity of each aircraft and mission. Air Combat Command officials told us that aircrews flying more complex combat aircraft may require additional training but stated that given the constrained resource environment that DOD and other federal agencies have operated under for the past several years, it would be difficult to increase training requirements and the flying hours required to meet them, even in the face of a potential need. According to its fiscal year 2017 budget request, the Air Force capped its flying-hour programs at a level it determined to be executable and therefore has not been able to grow readiness in the full range of missions over previous years. The budget request also notes that the fiscal year 2017 request is intended to align funding for the flying- hour program to maintain current readiness levels and to meet its highest priorities. However, Air Combat Command officials stated that the analysis needed to determine the relative differences in annual live-fly training requirements based on the complexity of various aircraft and missions had not been conducted.

In addition, the criteria for designating experienced and inexperienced aircrews are based on assumptions that have been in place for the last 40 years. The annual training requirements differ for aircrews designated as experienced and those designated as inexperienced. For example, inexperienced aircrews are required to complete one more training sortie each month than experienced ones. In most cases, aircrews are designated as experienced if they have flown 500 hours in their primary aircraft. Certain wing and squadron officials offered that, as a result of ongoing deployments to the Middle East, aircrews can achieve 500 hours in their primary aircraft on their first deployment. However, because the scope of missions these aircrews conduct while deployed is narrow, they are not necessarily getting training and experience across the full range of core missions. As a result, when they return from their deployment, these aircrews can be designated as experienced, and therefore they will not get the additional training that they may need. These officials also noted that aircrew capability within the squadron may suffer and there could be less capacity to fill squadron positions with sufficiently qualified aircrews.

Further, annual training requirements since fiscal year 2012 have set the same mix of live and simulator requirements across most of the platforms we reviewed. For all but one combat aircraft platform, both experienced and inexperienced aircrews are required to complete 36 simulator missions per year to be considered combat mission ready. These requirements were established based on the capacity of simulators at one installation and not on an analysis of the training needs of aircrews. According to Air Combat Command officials, the requirement for aircrews to fly 36 simulator missions per year was established based in part on the simulator capacity at Shaw Air Force Base, which had the greatest demands on its simulators because it supports three active fighter squadrons and an Air National Guard squadron. However, one wing commander and two squadron commanders told us that simulator requirements may need to differ among aircraft, depending on the full range of core missions that the aircraft is expected to perform. Additionally, Air Combat Command officials told us that aircrews of fifth- generation aircraft may require additional simulator training. In fact, the simulator training requirements for F-35 aircrews include one more simulator mission per month, for a total of 48 simulator missions per year. However, Air Combat Command did not provide any analysis that supports the need for an additional simulator mission for the F-35 or explain why the F-22, another fifth-generation aircraft, would not also require additional simulator training for its aircrews over what is required for fourth-generation aircraft. Based on our review of fiscal year 2015 data provided by Air Combat Command on available simulator hours at combat air forces bases, we found that there was excess simulator capacity at a number of locations. For example, F-15E squadrons at Mountain Home Air Force Base had an annual capacity of approximately 6,000 hours at their simulator facility, but the annual simulator requirement was approximately 4,000 hours.

According to our prior work on strategic training, training plans should be designed to determine the skills and competencies a workforce needs to achieve current, emerging, and future agency missions and to identify gaps—including those that training and development strategies can help address. Periodic reassessments should be part of a continual effort to evaluate and improve the agency’s training and development efforts, and evidence of timely changes should be reflected in those efforts. While Air Combat Command makes updates to the annual training plans each year, it has not comprehensively reassessed the assumptions underlying its annual training requirements—including, but not limited to, the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training—and updated these new assumptions in its training plans.

Officials at Air Combat Command told us that they had two studies under way to help inform the needed analysis to address these issues, but the studies had not been completed at the time of our review. For example, Air Combat Command has an ongoing study evaluating the current requirement that aircrews accomplish 500 flying hours in order to be considered experienced. The study is intended to explore various options for the F-22, including adjusting the current 500-hour requirement that is used to determine when to designate an aircrew as experienced. Air Combat Command officials stated that they may explore applying the results of this study to other combat aircraft. Additionally, in 2016, Air Combat Command issued a proposal to study the minimum, safe, and optimal numbers of live-fly and simulator training sorties aircrews must complete each year. The study is intended to include an examination of the minimum number of sorties required by mission type, the definition of proficient and familiar, and the mix of live and simulator training to support each mission set, among other areas. According to officials from Air Combat Command, the results of the study are expected to be available in 2017.

While these initiatives are a positive step, it is too early to determine how the results of the two studies will be used to implement changes to the Air Force’s annual training requirements or whether the scope of the studies is sufficient to address all of the assumptions that need to be revisited. Without fully reassessing the assumptions underlying its annual training requirements, the Air Force cannot be certain that its annual training plans are aligned with its stated goals to ensure a full-spectrum-capable force that can successfully achieve missions across a broad range of current and emerging threats.

Combat Fighter Squadrons Are Not Meeting All of Their Annual Training Requirements across the Full Range of Core Missions, and Completed Training Is Not Being Evaluated for Effectiveness

Combat Fighter Squadrons Are Not Meeting All of Their Annual Training Requirements across the Full Range of Core Missions

Based on our analysis of data on the completion of annual training, we found that combat fighter squadrons were generally able to complete mission training requirements for ongoing contingency operations, such as close air support to ground forces, but were unable to meet annual training requirements across the full range of core missions.

Various Factors Affected the Ability of Combat Fighter Squadrons to Complete Annual Training across the Full Range of Core Missions

Wing and squadron commanders cited various factors that have limited the ability of their squadrons to complete training across the full range of core missions. We analyzed the information we obtained from our interviews with four wing commanders and eight squadron commanders—including commanders overseeing aircrew training for both fourth- and fifth-generation aircraft—and found that they identified similar training limitations. They cited four common limitations: the maintenance unit’s ability to provide adequate numbers of aircraft for training, high deployment rates, adversary air tasking, and manpower shortfalls in the squadrons.

Seven of the squadron commanders we interviewed told us that their maintenance unit’s ability to provide an adequate number of aircraft affected their squadrons’ ability to complete their annual training requirements. These commanders stated that in recent years, maintenance units for U.S.-based squadrons have been consistently manned below their authorized levels. Air Force officials told us these shortfalls in maintenance personnel limit their ability to produce the number of aircraft required to meet annual live-fly training requirements.

Squadron commanders we interviewed also identified factors that can limit the time aircrews have available to meet annual training requirements, including the following: High deployment rates—Four squadron commanders told us that high deployment rates limited the ability of aircrews to conduct annual training across the full range of core missions between deployments. Certain fighter squadrons have been deploying every eighteen months for a 6-month period each time. During these 6-month deployments, squadrons have been flying primarily close air support missions and are unable to be proficient across their full range of core missions. When they return from deployment, squadrons have 12 months to rebuild proficiency across their full range of core missions before their next six-month deployment. Some squadron commanders told us that this high deployment rate typically reduces aircrews’ 12- month training schedule by at least 2 months, because they must focus on close air support training in preparation for the next deployment.

Adversary air tasking—Five squadron commanders told us that their squadrons have been tasked heavily to provide adversary air for large force exercises, such as joint and multinational “Red Flag” exercises designed to provide realistic training in a contested or degraded environment. According to Air Force officials, the demand for adversary air at large force exercises has increased because in fiscal year 2015 one of the two dedicated adversary air squadrons at Nellis Air Force Base was eliminated because of budgetary reductions. To help mitigate this reduction in adversary air capability, Air Combat Command increased the tasking of fighter squadrons to provide this capability. For example, an F-15E squadron commander told us that his squadron will have been tasked 3 of the first 9 months of fiscal year 2016 to provide adversary air for various exercises, including a Red Flag exercise. According to one squadron commander, participation as adversary air in these large force exercises can take up to 2 months out of the squadron’s training cycle to prepare for and execute this role. This limits the time available to conduct annual training for the squadron’s primary and secondary missions.

Squadron manpower shortfalls—Five squadron commanders told us that shortfalls in their squadron manning can limit the time aircrews have to meet their annual training requirements. Specifically, with fewer aircrews assigned to a squadron, aircrews are required to take on a larger share of the administrative functions of the squadron, including scheduling training and weapons evaluation. This challenge can be greater for squadrons operating under the total force integration construct, which uses aircrews from reserve component squadrons to supplement the aircrews in active-duty squadrons. For example, the two F-22 squadrons in the 1st Fighter Wing are integrated with F-22 aircrews from the 192nd Fighter Wing of the Virginia Air National Guard. As a result, there are fewer active-duty aircrews in the squadron. However, according to both F-22 squadron commanders, these aircrews are required to perform the administrative duties for the full squadron, because the Air National Guard aircrews are typically available only on the weekends, and their time with the squadron is used for training. Both F-22 squadron commanders we spoke with told us that even if maintenance personnel could generate the number of aircraft needed to meet their annual training requirements, the squadrons would be unable to conduct additional training, because their aircrews’ availability is limited by their other assigned responsibilities within the squadrons.

In addition to factors that limit the ability of squadrons to meet their annual training requirements, the squadron commanders we spoke with also cited a number of factors that affected the training that aircrews are able to accomplish. The extent to which these factors affected training differed between fourth- and fifth-generation aircraft. For fourth-generation aircraft, five squadron commanders told us that the considerable number of taskings to perform adversary air at large force exercises do not provide effective training opportunities. Additionally, some of these commanders expressed concern that adversary air taskings would increase and that recent changes to their annual training requirements may mask this problem. Specifically, prior to fiscal year 2016, the annual training requirements for certain combat aircrews set a cap on the annual number of adversary air sorties that could count toward their combat mission ready status. However, in the fiscal year 2016 training requirements, that cap was removed, allowing for all adversary air sorties to count toward the designation of combat mission ready. Two squadron commanders told us they were concerned that this change could result in masking the readiness ratings for squadrons, since overflying adversary aircraft sortie requirements provides limited training compared with their full range of core missions, but still allows the aircrews to be designated as combat mission ready. According to Air Combat Command officials, this change was made to encourage aircrews to accurately log adversary air sorties. These officials stated that previously some aircrews may have been flying in an adversary air role, but would complete some other limited type of training maneuver so that they could record the sortie as a different mission that would count toward their combat mission ready status.

For fifth-generation aircraft, all three squadron commanders we interviewed cited various factors affecting training, including airspace and training range limitations and the availability of adversary air to fly against. For example, two F-22 squadron commanders at Langley Air Force Base told us that the size of the base’s airspace limits their ability to train for their more complex missions, including offensive counter air and defensive counter air missions, because the available airspace constrains the capabilities they are able to deploy. Additionally, all three of the commanders for squadrons flying fifth-generation aircraft told us that limits in training range capabilities, such as threat replicators and targets, affected the training completed at smaller regional training ranges, as well as at larger training ranges such as the Utah Test and Training Range and the Nevada Test and Training Range. According to these officials, the training ranges lack many of the more advanced threat replication systems that can challenge F-35 and F-22 capabilities and provide effective training across their full range of core missions. According to training range officials at the Nevada Test and Training Range, the current range capabilities are adequate to replicate only a “first Gulf War”- era threat scenario.

Air Combat Command has implemented some initiatives to help increase the number of training sorties that aircrews are able to conduct. However, these initiatives can further affect the training that is completed. For example, Air Combat Command has received increased funding for “Red Flag” exercises to enable a fourth annual exercise at Nellis Air Force Base. These exercises can provide a high-end, realistic training environment for aircrews. However, officials explained that Red Flag exercises require a significant adversary air presence to provide high quality training for the participants. As discussed above, these exercises can result in squadrons being overtasked to fly as the adversary force, which can further limit aircrews’ availability to train to their full range of core missions. To help mitigate the burden on squadrons, in December 2015, the Air Force began contracting with a private company to provide adversary air capabilities on a short-term basis to support exercises at Nellis Air Force Base. According to Air Force officials, in fiscal year 2017, the Air Force plans to spend about $25 million to contract for six A-4 aircraft (a legacy fighter platform first developed in the 1950s) to provide adversary air for two months of exercises.

In addition, squadron commanders, in coordination with maintenance groups, have developed some initiatives to increase the number of sorties they are able to produce, but these initiatives can further limit the training that is completed. For example, seven squadron commanders told us they have implemented the use of “hot pits”—a practice where the aircrew flies a sortie, lands and refuels, and then flies another sortie without shutting down the aircraft. This practice minimizes the aircraft repairs that may arise between sorties if an aircraft is shut down and then restarted. Five squadron commanders told us that they also occasionally use tanker aircraft to refuel their squadrons in the air during training. Both “hot pitting” and aerial refueling practices produce an additional sortie for the aircrews that day. However, squadron commanders told us that such practices prevent the aircrews from getting the full benefit from both training sorties. Specifically, these practices do not allow the aircrews to receive a mission brief and debrief for both sorties, a critical learning component of effective training. In addition, squadron commanders told us that without the benefit of a full briefing for the second training sortie, the aircrews are not able to conduct their more complex missions that require additional instruction to be effective. Figure 5 shows an F-22 aircraft conducting a “hot pit” refueling and an F-15C aircraft conducting an aerial refueling.

The Air Force has a number of efforts under way to study or address some of the factors that limit the ability of fighter squadrons to meet annual training requirements and that affect the training that aircrews are able to accomplish, but it will likely take time before the full benefit of these efforts can be assessed. For example, the Air Force has announced several initiatives to manage the shortage of maintenance personnel, including hiring additional maintenance personnel, temporarily transitioning active-duty maintenance units from some legacy aircraft, and offering selective reenlistment bonuses as an incentive to improve retention, among other initiatives. According to Air Force officials, it may take several years before newly hired maintenance personnel will have the training and experience they need in order to improve aircraft availability rates. In addition, Air Combat Command officials told us that there is an ongoing study reviewing adversary air requirements. Further, the Air Force has taken steps to upgrade its training infrastructure, including training ranges. For example, in its fiscal year 2017 budget request, the Air Force requested a funding increase of about $35 million for the maintenance and sustainment of 21 training ranges, including installation of additional threat emitters and expansion of communication capabilities between air and ground forces.

The Air Force Does Not Systematically Evaluate the Effectiveness of Completed Training against Expectations

Air Force processes used to record and monitor combat fighter squadrons’ annual training are focused on the frequency of training completed to meet annual requirements and do not include a systematic evaluation of the effectiveness of that training against established expectations. Squadron commanders track and report monthly and 3- month “look-backs” to determine if aircrews are meeting their prescribed annual training requirements. These assessments emphasize the frequency of training and inform the designation of aircrews as combat mission ready. According to Air Force officials, squadron commanders are delegated with the authority to determine whether a sortie should “count” toward meeting the annual training requirements. These determinations are based on Air Force guidance—including AFI 11-2, Volume 1—that establish common training standards for each type of combat aircraft and documentation supporting the Ready Aircrew Program that provides a list of training expectations that include the types of events that should be included in a training mission to be considered effective. For example, the Ready Aircrew Program tasking memorandums establish when a non-effective sortie should be recorded, for example when training limitations are caused by weather conditions, aircraft maintenance or weapon malfunctions, or cancellation of training support elements—which can include adversary air or air-refueling assets.

Air Combat Command uses quantitative data on the completion of annual training requirements and qualitative information on broad factors that affect training in various forums that discuss training. However, our review of documentation supporting these forums found that the Air Force does not systematically evaluate the extent to which completed training met expectations or the degree to which certain challenges have affected training. In our review, we identified two forums that Air Combat Command uses to oversee training completion and discuss training challenges. These forums are focused on a review of training completion data and include discussions of factors that affect training completion and the effectiveness of training. For example, at the end of each fiscal year, Air Combat Command aggregates combat fighter squadrons’ training completion data for its end-of-year training and readiness review. Tables are compiled by squadron and by aircraft type to provide a comparison of annual training completion data against each aircraft’s full range of core missions. According to Air Combat Command officials, the squadron commanders are required to submit a list of qualitative factors that, in general, affected the completion and effectiveness of the squadron’s training. These factors are then discussed at an end-of-year training and readiness review, which is an informational briefing used to inform the state of readiness of the combat air forces.

In addition, the annual Realistic Training Review Board provides another forum for discussion between Air Combat Command and leadership at the group commander level on training challenges. According to officials and our review of documentation discussed during these forums, these discussions are focused on a number of issues that affect training, such as funding for the flying-hour program and distributed mission operations. Further, the Board identifies a number of action items related to training that need to be addressed. For example, output from the 2015 Board included actions directed toward improving cruise missile threat replication, targets for F-22 ranges, and opportunities for certain weapons employment for F-15C squadrons. While these forums provide an opportunity for Air Force officials to discuss and identify training challenges, neither includes a systematic evaluation of the effectiveness of training against established expectations in order to identify limitations to the effectiveness of training.

The lack of a systematic evaluation of the effectiveness of training against established expectations has affected the Air Force’s ability to make informed resource decisions to improve its training infrastructure. For example, training range officials from both the Nevada and Utah Test and Training Ranges told us that without information on the gaps between aircraft training requirements and current range capabilities, they were not well positioned to make informed decisions on where to focus their resources to improve range capabilities. An official from the Nevada Test and Training Range told us that identifying training gaps is becoming even more important, because the new F-35 aircraft will have additional range requirements that exceed those of fourth-generation fighter aircraft. However, under current Air Force guidance, the Nevada Test and Training Range is required to provide capabilities only for large force exercises and does not currently possess the range capabilities needed to support a number of training events identified in the training expectations for the F-35, including close air support in an urban terrain and certain attack events, among others.

We found that the Air Force does not systematically evaluate the effectiveness of training against established expectations for two reasons. First, Air Combat Command has not established the desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums. Our prior work on strategic training notes that agencies should systematically evaluate the effectiveness of their training efforts and collect data corresponding to established training objectives to continually improve, deliver, and enhance training and ensure the effective allocation of resources. In contrast to the Ready Aircrew Program tasking memorandums, we found that documentation underpinning other types of aircrew training, such as mission qualification training and upgrade training, provide more detailed information on critical features needed to measure the effectiveness of training sorties. For example, the upgrade training syllabus for the F-22 provides desired learning objectives for an aircrew to conduct an effective defensive counter air sortie and information on the number of adversary aircraft required and the period of time that should be spent defending a target. However, the Ready Aircrew Program tasking memorandum for the F-22 does not include similar information on an effective defensive counter air sortie. Instead, it provides only a list of expectations that could be included in a training mission, along with information on the desired amounts of adversary air needs.

Second, the current process used by squadrons to report information on their training completion does not collect data on the effectiveness of the training against established expectations. Documentation supporting the Ready Aircrew Program notes that credible and detailed training accomplishment reporting provides an accurate assessment of readiness and the data needed to advocate for adequate resources. This documentation notes that Air Combat Command expects to see data that document the underlying challenges to providing an effective training program. However, the form that aircrews are required to complete following a training sortie requires them to record the number of flying hours and other information, such as the type of mission, but does not capture information on the ability of aircrews to meet the training expectations established in the Ready Aircrew Program. For example, the form does not provide the opportunity to record the training support elements involved in the sortie, such as the number of adversary aircraft involved and the threat replicators available. In the absence of a process to obtain this information, squadron commanders are also unable to collect data corresponding to the effectiveness of annual training completed to meet expectations or the degree to which certain challenges affected training. One squadron commander we interviewed has developed internal guidance for aircrews to determine whether a sortie was effective and should count toward the annual training requirements. This commander provides an annual guidance letter to be used within the squadron that sets the minimum requirements that must be met to log an effective training sortie. However, based on our interviews with other squadron commanders, this practice is not being consistently performed across all squadrons.

Officials from Air Combat Command told us that they had not developed more detailed expectations for combat fighter squadron commanders to measure and collect data on the effectiveness of annual training for several reasons. These officials stated that Air Force instructions and the Ready Aircrew Program provide squadron commanders with the authority to determine if a training sortie was effective and should be counted toward meeting annual training requirements. Further, these officials noted that aircrews have been limited in their ability to complete annual training requirements due to the factors previously discussed in this report and that having more detailed expectations for measuring the effectiveness of annual training could make it more difficult for aircrews to achieve a combat mission ready status. In addition, Air Combat Command officials stated that the need to complete training against more detailed expectations and collect information on training effectiveness should be balanced against the time and other factors that would be required to collect the information.

Given the range of factors that have limited the ability of combat fighter squadrons to complete annual training requirements, having a consistent basis for monitoring training results is critical in tracking progress toward achieving the Air Force’s goal of training units for the full range of core missions. Moreover, Air Combat Command has stated its goal of moving to a training model based on the effectiveness of training, using quantifiable levels of performance, rather than the current model focused on the frequency of training. Without a process to systematically measure and collect data on the effectiveness of training against established expectations, the Air Force will not be well positioned to improve, deliver, and enhance training or allocate resources to address factors that limit the effectiveness of training.

Air Force Planning Documents for Virtual Training Do Not Include All Desirable Characteristics of a Comprehensive Strategy

The Air Force’s virtual training plans do not include all desirable characteristics of a comprehensive strategy, such as a risk-based investment strategy or a time line for addressing training needs. We have previously reported that when addressing national or department-wide issues, it is standard practice for organizations to have a strategy that includes desirable characteristics such as establishing goals and objectives, identifying actions for addressing those objectives, allocating resources, identifying roles and responsibilities, and measuring performance against objectives, among others.

In recent years, the Air Force has developed some planning documents to guide its virtual training efforts. In 2013, for example, the Headquarters Air Force issued its United States Air Force Live Virtual Constructive Operational Training Flight Plan. The flight plan was the first Air Force level strategic document to address LVC operational training from an Air Force-wide perspective. The document established the Air Force’s broad vision for the future of the LVC operational training environment and highlighted areas that needed attention to advance the program and realize its full potential. In addition, in 2014, Air Combat Command issued the Combat Air Force Live, Virtual, Constructive Vision, which describes the command’s goals for the integration of LVC operational training capabilities, along with the priority tasks and general timeframes for implementation. Further, in 2015, Air Combat Command released the Combat Air Force Live, Virtual, Constructive Roadmap that sets near-, mid-, and far-term tasks that are required to better define needed training capabilities.

While the Air Force has issued these planning documents to help guide its LVC operational training initiatives for the combat air forces, they do not include all of the characteristics we have identified as being desirable in comprehensive strategies. Based on our analysis of the United States Air Force Live Virtual Constructive Operational Training Flight Plan and the Combat Air Force Live, Virtual, Constructive Vision, we found that the Air Force has only partially addressed or has not addressed most of the desirable characteristics, as shown in table 1.

For one characteristic, we found that both of the planning documents contained the desirable elements. For example, United States Air Force Live Virtual Constructive Operational Training Flight Plan establishes the roles and responsibilities for organizations involved in the implementation of LVC operational training initiatives, including Headquarters Air Force and the major commands, such as Air Combat Command. It also discusses the governance structure and the role of primary support organizations, such as the LVC Council of Colonels and the Distributed Warfare Integration Cell, which are intended to identify and prioritize LVC operational training gaps and shortfalls that cut across the major commands. For other characteristics, we found that some of the elements were included, but not all. For example, the Combat Air Force Live, Virtual, Constructive Vision identifies a number of implementation tasks across a range of implementation phases, but it does not establish priorities, specific milestones, or performance measures. Our prior work has shown that these elements can help to ensure effective implementation and accountability to achieve results. Further, for some characteristics, we determined that neither planning document contained any of the desirable elements. For example, neither of the two documents identified what the virtual training efforts will cost, the sources and types of resources required, or the prioritization of needed investments. Our prior work has shown that a risk-based investment strategy may be used to define and prioritize related resource and operational requirements, as well as to develop a time line for obtaining capability needs. A risk-based investment strategy includes five key phases: (1) setting strategic goals and objectives and determining constraints, (2) assessing risks, (3) evaluating alternatives for addressing these risks, (4) selecting the appropriate alternatives, and (5) implementing the alternatives and monitoring the progress made and results achieved.

We identified several areas where the Air Force is making significant investments in virtual training, despite the lack of a comprehensive strategy needed to guide and prioritize these efforts. In some cases, this has led to investments that may not maximize available Air Force resources. For example, the Air Force is building an F-22 simulator facility at Nellis Air Force Base. The facility is expected to be completed in fiscal year 2017, at an estimated cost of $14 million. However, according to officials at Nellis Air Force Base, the Air Force does not expect the four simulators to arrive there until fiscal year 2019 at the earliest, leaving the facility potentially unutilized during that interim period. Additionally, officials identified a number of examples where the Air Force has not made cost-efficient investments in virtual training systems. For instance, officials from Headquarters Air Force noted that different versions of the F-16 simulator have been procured within the active, guard, and reserve components, which create inefficiencies among the components as they develop, procure, and sustain these differing simulators.

Air Force officials also provided examples of prior investments in simulators that do not meet the training needs of aircrews across their full range of core missions. For example, the Air Force Audit Agency reported in 2013 that Air Force officials had not adequately defined F-22 simulator requirements, because the procurement contracts did not include simulator performance specifications. Instead, all F-22 simulators shared a single training system specification that provided insufficient technical detail as to what the Air Force considered an acceptable product. As a result, F-22 squadrons at Langley Air Force Base have traveled to the Air Combat Simulator facility in Marietta, Georgia to utilize the contractor- owned F-22 simulators at that location. Officials from the F-22 squadrons told us that the contractor-owned simulators provide higher-quality training than the simulators at their home station and allow them to train in a more realistic threat environment. For 2015, the cost of using the Air Combat Simulator facility in Marietta was around $10,000 per day, plus travel costs for the aircrews. According to an official from Air Combat Command, F-22 squadrons at Langley Air Force Base utilized the Air Combat Simulator facility for 5 days in calendar year 2015 and, as of February 2016, have utilized the facility for 11 days in calendar year 2016. In addition, based on our review of simulator utilization data, the F- 22 simulators at Langley Air Force Base remain underutilized.

Air Combat Command has identified significant requirements for LVC operational training capabilities that will require funding over the next several years. Specifically, the Air Force estimates that LVC operational training requirements will total approximately $3.8 billion in funding for fiscal years 2015 through 2019. This includes funding to maintain simulator currency with the aircraft, modernize simulator visual systems, secure virtual training networks, and hire additional simulator instructors. Air Force officials told us that concern over simulator information security—particularly for fifth-generation aircraft—will require network upgrades to ensure that the infrastructure has adequate controls in place to protect data. According to these officials, network security and information assurance are currently the most pressing challenges in the Air Force’s virtual training environment.

While the United States Air Force Live Virtual Constructive Operational Training Flight Plan and the Combat Air Force Live, Virtual, Constructive Vision demonstrate some effort toward defining a broad strategy for the Air Force’s virtual training efforts, the Air Force lacks a comprehensive strategy, because it has not fully refined its planning documents to achieve its training vision, for example by developing a risk-based investment strategy that identifies and prioritizes capability needs and includes a time line for addressing them. The Flight Plan recognizes the need for an investment strategy to prioritize and guide investments in future LVC operational training capabilities; however, officials told us that development of an investment strategy has not begun. Additionally, as we reported in 2012, the Air Force does not have a methodology to fully account for and track the costs of its virtual training enterprise. Headquarters Air Force officials told us that they have made some progress in tracking operation and maintenance costs for virtual training but none toward tracking investment funding. As we reported in 2012, identifying virtual training costs is challenging, because funds that support virtual training are dispersed across multiple program elements. Headquarters Air Force officials told us that they have identified approximately 75 program elements that support the LVC operational training program, but that it is difficult to determine if these elements represent all required funding. Further, these officials told us that these 75 program elements are not consistently monitored, because the automation required to monitor and conduct any trending analysis on the program elements does not currently exist.

Given the importance of an integrated LVC operational training environment to the future of Air Force training, a comprehensive strategy would help ensure that the Air Force develops the capabilities needed to meet its virtual training needs. Without a comprehensive approach to virtual training that includes a risk-based investment strategy and a time line for prioritizing and addressing capability needs, the Air Force may be slow to develop needed capabilities and therefore could increase risk for fifth-generation fighter aircraft, whose training will require a robust, realistic virtual training environment.

Conclusions

The Air Force has identified the need to ensure a full-spectrum capable force that can successfully achieve missions to address a broad range of current and emerging threats. However, the Air Force has not completed the analysis and planning needed to ensure that aircrew training requirements fully align with those goals. Specifically, Air Combat Command has not comprehensively reassessed the assumptions underlying its annual training requirements, some of which have been in place for 40 years. Without fully reassessing the assumptions underlying its requirements, the Air Force cannot be certain that its annual training plans are aligned with its stated goals to ensure a full-spectrum capable force that can successfully achieve missions across a broad range of current and emerging threats. In addition, Air Combat Command has not identified desired learning objectives or training support elements necessary to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums. Having a consistent basis for monitoring training results is critical for tracking progress in achieving the Air Force’s goal of training units for the full range of core missions. Also, without a process to collect data to assess the effectiveness of the training, the Air Force may not be well positioned to improve, deliver, and enhance training or allocate resources to address factors that limit the effectiveness of training. Further, the Air Force lacks a comprehensive strategy for virtual training, because it has not fully refined its planning documents to achieve its training vision, for example by developing a risk- based investment strategy and a time line for prioritizing its needs. Achieving this vision is critical; as new fifth-generation aircraft become operational they will require a robust and realistic virtual training environment to train to their full capabilities.

Recommendations for Executive Action

To ensure that annual training plans are aligned with the Air Force’s stated goals to ensure that its forces can successfully achieve missions across a broad range of current and emerging threats, we recommend that the Secretary of Defense direct the Secretary of the Air Force to comprehensively reassess the assumptions underlying its annual training requirements—including, but not limited to, the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training—and make any appropriate adjustments in future training plans.

To improve the Air Force’s ability to consistently monitor training results and better position it to allocate resources to address factors that limit the effectiveness of training, we recommend that the Secretary of Defense direct the Secretary of the Air Force to establish desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums and develop a process to collect data to assess the effectiveness of annual training against these features.

To improve the Air Force’s ability to develop the capabilities needed to meet its virtual training needs, we recommend that the Secretary of Defense direct the Secretary of the Air Force to continue to refine its planning for virtual training to incorporate the desirable characteristics of a comprehensive strategy, including developing a risk-based investment strategy that identifies and prioritizes capability needs and includes a time line for addressing them.

Agency Comments and Our Evaluation

In written comments on the non-public sensitive version of this report, DOD did not concur with the first and second recommendations, and concurred with the third recommendation. DOD’s comments are reprinted in their entirety in appendix II.

In its comments, DOD stated it did not concur with our first recommendation, but DOD identified some actions it was taking to address the intent of the recommendation. Specifically, DOD stated that the Air Force has already begun a comprehensive reassessment of the assumptions underlying its annual training requirements and that those results will be used to influence future training plans. More specifically, DOD stated that the Air Force has directed a study that will reassess the full spectrum of the Air Force’s Ready Aircrew Program training requirements and provide a range of readiness levels associated with varying training levels. According to DOD’s comments, the estimated completion date of the study is September 2017. DOD also stated that a second study is reviewing the factors that contribute to the Air Force’s aircrew experience designation. For example, the Air Force has completed a review of the F-22 that resulted in substantive changes for the F-22 experiencing criteria. DOD’s comments further stated that the Air Force has directed a follow-on study for all fourth-generation fighter aircraft with an estimated completion date for this study of September 2017. As we noted in our report, the Air Force’s initiatives represent positive steps to study aspects of the combat aircrew annual training requirements. However, given that these studies are in the initial stages or not yet underway, it is unclear whether the scope of the studies is sufficient to address all of the assumptions that need to be revisited. Further, it is too early to determine what adjustments, if any, the Air Force will make in its future training plans in response to the studies’ results, as we recommended. Therefore, we believe the recommendation remains valid.

DOD did not concur with our second recommendation, stating that the Air Force’s Ready Aircrew Program training differs significantly from other syllabus-directed courses of instruction and that desired learning objectives for this training are set at the squadron level in accordance with current Air Force guidance. Specifically, the Ready Aircrew Program tasking memorandums direct squadron commanders to develop unit training programs that focus on their primary and secondary missions, and that in order to be effective, each mission must successfully complete a sufficient number of events applicable to that mission type, according to the squadron commander. In addition, DOD’s comments noted that Air Force guidance provides higher headquarters direction at the appropriate level while providing local commanders with the flexibility to set detailed learning objectives for individual sorties that allows them to maximize the training value and efficiency of each sortie. DOD also stated that a table in the Ready Aircrew Program tasking memorandums identifies the desired adversary air ratios, which serves as the training support elements needed to accomplish training expectations. Finally, DOD stated that because the effectiveness of annual continuation training is identified at the local level, only the accomplishment of training needs to be collected and reported. We agree that squadron commanders should have the flexibility to inform individual aircrew training needs. However, establishing a baseline of desired learning objectives to meet Air Combat Command’s training expectations for the Ready Aircrew Program will provide squadron commanders information on what features a training event should include and help ensure that aircrews are receiving consistent and comparable training across the force. Further, while the Ready Aircrew Program tasking memorandums identify desired adversary air ratios, the memorandums do not identify any other training support elements that are needed to accomplish training expectations. For example, the tasking memorandums do not provide information on the training range capabilities needed to meet expectations, such as threat replicators and targets. As noted in our report, a range of factors, in addition to the availability of sufficient ratios of adversary air, have limited the ability of combat fighter squadrons to complete annual training requirements. The lack of this information can affect the Air Force’s ability to make informed resource decisions to improve its training infrastructure. Finally, our report also noted that Air Combat Command guidance states that it expects to see data that documents the underlying challenges to providing an effective training program. DOD’s comment that only the accomplishment of training needs to be collected and reported contradicts this guidance as well as the Air Force’s stated goal of moving to a training model based on the effectiveness of training, using quantifiable levels of performance, rather than the current model focused on the frequency of training. Without developing a process to collect data and measure the effectiveness of training against established expectations that includes information on desired learning objectives and training support elements, the Air Force will not have the information required to achieve this goal. Therefore, we believe the recommendation remains valid.

In concurring with the third recommendation, DOD stated that the Air Force is beginning to take a holistic view of operational training and the Air Force has stood up an Operational Training Division to better focus control over the operational training enterprise. One responsibility of the division will be to update the United States Air Force Live Virtual Constructive Operational Training Flight Plan with all of the desirable characteristics of a comprehensive strategy. The Air Force expects to complete this update by September 2017. By updating this plan to include the desirable characteristics of a comprehensive strategy, such as a risk- based investment strategy that identifies and prioritizes capability needs and a time line for addressing them, the Air Force would improve its ability to develop the capabilities required to achieve its virtual training needs.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, and the Secretary of the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report are to determine the extent to which the Air Force has (1) determined requirements to train its combat aircrews for the full range of core missions, (2) met annual training requirements for combat fighter squadrons across the full range of core missions and evaluated the effectiveness of this training, and (3) established virtual training plans that include desirable characteristics of a comprehensive strategy.

This report is a public version of the prior sensitive report that we issued in August 2016. The Department of Defense (DOD) deemed some of the information in the prior report as For Official Use Only (FOUO), which must be protected from public disclosure. Therefore, this report omits FOUO information and data on some of the Air Force’s training priorities, completion of annual training requirements for active-duty fighter squadrons, and aircraft maintenance generation capabilities. Although the information provided in this report is more limited in scope, it addresses the same objectives as the sensitive report. Also, the methodology used for both reports is the same.

We focused our review on the annual continuation training requirements for aircrews assigned to aircraft within the active component of the combat air forces. We focused on Air Combat Command as the lead major command for all combat air forces, which has responsibility for developing and managing annual continuation training requirements through the Ready Aircrew Program. The aircraft included in our review were the A-10, F-15, F-16, F-22, and F-35. We selected a non- generalizable sample of units to speak with based on our analysis of annual training completion data and the range of units’ core missions. Based on this analysis, we conducted site visits with F-22 units at Langley Air Force Base, F-16 Block 40 and F-35 units at Hill Air Force Base, F- 15E units at Seymour Johnson Air Force Base, and F-16 Block 50 units at Shaw Air Force Base.

To determine the extent to which the Air Force has determined requirements to train its combat aircrews for the full range of core missions, we analyzed Air Force documents on training to identify the Air Force’s current process for developing training requirements, including the Air Force Instruction 11-2, Volume 1 for each aircraft in our review. We interviewed officials involved in the development of the annual training requirements, including officials from Air Combat Command, U.S. Air Forces Europe, Pacific Air Forces, and U.S. Air Forces Central Command. We also interviewed wing and squadron commanders at selected units, as described above, to obtain an operational-level perspective on the training requirements needed to meet the full range of core missions. We reviewed the annual training requirements established for each aircraft for fiscal years 2012 through 2016 to determine how the Air Force establishes the experienced and inexperienced aircrew designations and how sortie and simulator mission requirements have changed over time. We evaluated these processes against Department of Defense (DOD) and Air Force guidance that prioritize and establish training requirements and against the leading practices we have identified for managing strategic training.

To determine the extent to which the Air Force has met annual training requirements for combat fighter squadrons across the full range of core missions and evaluated the effectiveness of this training, we analyzed training completion data from fiscal years 2012 through 2015 to identify core missions for which aircrews were or were not fully trained. We performed data reliability procedures on the information included in Ready Aircrew Program completion reports by comparing that information against related documentation and interviewing knowledgeable officials on controls over the reporting systems. We determined that the data presented in our findings were sufficiently reliable for the purposes of this report. We reviewed Air Force documentation that describes training challenges and interviewed Air Force officials, including wing and squadron commanders at selected units, as described above, and officials at Air Combat Command’s two major range and test facility bases—the Nevada Test and Training Range and the Utah Test and Training Range—to discuss any factors that limit the ability of aircrews to complete training for the full range of core missions. We evaluated the Air Force’s process for assessing the effectiveness of its annual training against leading practices we identified for managing strategic training and against the Ready Aircrew Program tasking memorandums, which specify annual continuation training for personnel assigned to combat units.

To determine the extent to which the Air Force has established virtual training plans that include desirable characteristics of a comprehensive strategy, we reviewed planning documents established by the Air Force to guide its virtual training efforts and evaluated these actions against our prior work on desirable characteristics of national strategies. Given that there is no established set of requirements for strategies, we relied on our assessments of national strategies and the criteria that were applied to assess these strategies. We identified six desirable characteristics that national or department-wide strategies should contain. We assessed these criteria against the two existing Air Force planning documents we found most relevant to the combat air forces—United States Air Force Live Virtual Constructive Operational Training Flight Plan and the Combat Air Force Live, Virtual, Constructive Vision—to determine the extent to which these documents contain the elements of a comprehensive strategy. We determined whether these actions addressed, partially addressed, or did not address the desirable characteristics of strategy documents, based on our assessment of whether the planning documents explicitly cited all elements of a characteristic and the level of specificity and detail included.

We conducted this performance audit from June 2015 to September 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, Matthew Ullengren, Assistant Director; Karyn Angulo; Russell Bryan; Joanne Landesman; Kirsten Lauber; Amie Lesser; Jim Lloyd; Justin Riordan; and Sabrina Streagle made key contributions to this report.