Air Force Training: Actions Needed to Better Manage and Determine Costs of Virtual Training Efforts
Published: 2012-07-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Over the last 20 years, the Air Force has sought ways to expand its approaches to meeting aircrew training requirements, including the increased use of virtual training. In 2012, the Air Force reduced live flying hours, which it estimates will save $1.7 billion in fiscal years 2012 through 2016, as part of its response to the Secretary of Defenses efficiency initiatives. GAO conducted this study in response to House Report 112-78, accompanying a bill for the Fiscal Year 2012 National Defense Authorization Act, which directed GAO to review the status of the military services virtual training programs. Specifically, GAO assessed (1) how the Air Force determines the mix of live and virtual training to meet training requirements; (2) the extent to which the Air Force has an overarching organizational framework to guide, oversee, and integrate its virtual training efforts; and (3) the extent to which the Air Force considered costs related to virtual training in estimating potential savings from its training efficiency initiative. To do so, GAO analyzed guidance and other documents, visited virtual training facilities, and interviewed officials from the Office of the Secretary of Defense, the Joint Staff, and the Air Force.
section_title: What GAO Found
paragraphs: The three lead Air Force major commandsAir Mobility Command, Air Force Special Operations Command, and Air Combat Commandall utilize training requirements review boards composed of subject-matter experts to determine training requirements for specific aircraft. These boards determine which training requirements can be completed in live or virtual environments based upon factors such as specific combatant command mission requirements and the capabilities of fielded simulators and networks. All three commands use a combination of live and virtual approaches, but the mix varies by aircraft. For example, Air Combat Command specifies that approximately 25 percent of its training requirements could be met virtually. The other two commands conduct approximately 50 percent of their training virtually.
The Air Force has taken steps to manage its virtual training efforts, but its approach lacks some key elements of an overarching organizational framework needed to fully integrate efforts and address challenges. It has reorganized offices and undertaken various initiatives intended to enhance existing virtual training capabilities, but has not designated an entity to integrate these efforts or developed an overarching strategy to define goals, align efforts, and establish investment priorities. As a result, major commands have developed their own investment plans and standards for acquiring and fielding virtual training systems, which are often not interoperable and require costly, time-consuming work-arounds to allow personnel to train together and with joint and coalition partners. GAOs prior work has found that a designated entity with the necessary authority and resources and an overarching strategy are critical elements of managing organizational transformations and meeting long-term goals and agency missions. In the absence of an approach that establishes clear accountability and a strategy to guide its planning and investment decisions, the Air Force will continue to be challenged to guide the efforts of its commands in planning for and investing in virtual training, ensure these efforts meet the highest priority needs and are synchronized to avoid gaps or future interoperability issues, and maximize available resources.
The Air Force estimated it could save about $1.7 billion in its training program by reducing live flying hours and taking other steps, such as increasing the use of virtual training, but it lacks a methodology for determining the costs of virtual training and therefore did not consider these costs in its estimate. The Air Force estimated savings based solely on reductions in live flying hours without considering expenses such as those incurred for aircrew to travel to simulators, contractor personnel to schedule and operate simulators, and purchase of additional simulators. GAO has found that decision makers need visibility over financial data to meet agency goals and effectively use resources. Identifying virtual training costs is challenging because data is spread across multiple program elements in the Air Forces accounting structure. The Air Force completed an initial study in September 2011 that identified some costs related to virtual training, but it concluded these data might not be complete. In the absence of taking further steps to determine the universe of costs and a means to collect and track data, the Air Force will be limited in its ability to make fully informed investment decisions about the mix of live and virtual training in the future.
section_title: What GAO Recommends
paragraphs: GAO recommends that the Air Force designate an entity to integrate its virtual training efforts, develop a strategy to align virtual training initiatives and goals, and develop a methodology to collect virtual training cost data. DOD concurred with GAOs recommendations and identified planned actions.

section_title: Letter
paragraphs: Over the past two decades, the Air Force has sought ways to move  beyond the use of live flying to meet aircrew training requirements. One  alternative it has pursued is to increase its use of virtual training, which  utilizes simulators and information networks to bring together  geographically separated units in order to conduct training. Virtual  training allows personnel to replicate required training interactions and  procedures while reducing fuel costs and avoiding other constraints that  can affect live training, such as limited access to ranges. Air Force  leadership considers virtual training to be the cornerstone of its training  transformation efforts, and the Air Force expects to rely even more  heavily on virtual training as it fields new aircraft. Further, among the  initiatives developed in response to the Secretary of Defense’s direction  to find cost savings through increased department-wide efficiency, the Air  Force reduced its fiscal year 2012 budget for its flying hour program,  which funds live training, by $268 million while calling for an increase in  the use of virtual training.
H.R. Rep. No. 112-78 (2011), which accompanied a bill for the National  Defense Authorization Act for Fiscal Year 2012, directed us to review the  status of the military services’ training programs and report the results to  the House and Senate Armed Services Committees.assessed (1) how the Air Force determines the mix of live and virtual  training to meet training requirements (2) the extent to which the Air  Force has an overarching organizational framework to guide, oversee,  and integrate its virtual training efforts; and (3) the extent to which the Air  Force considered costs related to virtual training in estimating potential  savings from its training efficiency initiative. We are reporting separately  on the other services.
For this review, we  To address these objectives, we met with officials from the Office of the  Secretary of Defense, Joint Staff, Office of the Secretary of the Air Force,  Headquarters Air Force, and several Air Force major commands, and  visited virtual training facilities. Our review focused primarily on virtual  training systems for manned aircraft from combat air forces, mobility air  forces, and special operations forces. To determine how the Air Force  determines the mix of live and virtual training, we analyzed Air Force  studies and assessments of virtual training technologies and capabilities.  We also interviewed officials from several major commands and obtained  and analyzed training-requirement instructions for combat, mobility, and  special operations aircraft. To determine the extent to which the Air Force  has developed an overarching organizational framework to guide,  oversee, and integrate its virtual training efforts, we met with officials from  the above organizations as well as officials from Joint and Coalition  Warfighting, Joint Training Integration and Evaluation Center, Navy Air  Warfare Center, and the four primary centers that facilitate distributed  mission operations. To determine the extent to which the Air Force  considered costs related to virtual training in estimating potential savings  from its training efficiency initiative, we obtained and analyzed the Air  Force efficiency calculation and compared it with cost-saving estimating  best practices. We also interviewed officials from the Air Force Defense  Contracting Management Organization and Headquarters Air Force  budget office, as well as officials from the major commands listed above.  More-detailed information about our scope and methodology is provided  in appendix I.
We conducted this performance audit from August 2011 to July 2012 in  accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: For each of their aircraft, the Air Force lead commands set training  requirements that aircrews must complete on an annual basis in order to  maintain combat mission readiness. These training requirements include  basic tasks such as take-offs and landings and also more-advanced  tasks, such as air-to-air combat and ground-attack missions. To help  meet these requirements, the Air Force has developed an approach to  training that it terms distributed mission operations. This approach is  intended to train units as they expect to fight, maintain readiness, and  conduct mission rehearsals in a realistic environment. Distributed mission  operations utilizes the integration of virtual (e.g., a person training in a  simulator) and constructive (e.g., computer generated) elements to train  personnel at geographically separated sites by means of a network. For  the purposes of this report, we refer to training that includes a simulator  as virtual training.
The Air Force has four primary centers that facilitate distributed mission  operations by connecting units and simulators from geographically  dispersed areas:
Distributed Mission Operations Center in Albuquerque, New Mexico,  managed by Air Combat Command;
Distributed Training Operations Center in Des Moines, Iowa,  managed by the Air National Guard;
Warrior Preparation Center in Einsiedlerhof, Germany, managed by  U.S. Air Forces Europe; and
Korean Air Simulation Center in Osan, Republic of Korea, managed  by Pacific Air Forces.
The Distributed Mission Operations Center functions as the lead  integrator of virtual systems to conduct theater-level exercises and events  that include air, land, space, cyber, and maritime virtual assets for Air  Force, joint, and coalition partners. These large-scale events, known as  virtual flags, are conducted quarterly and last about 2 weeks. In 2011, the  Distributed Mission Operations Center trained over 1,400 personnel from  the Air Force, Army, Navy, Marines, and coalition forces through this  virtual exercise as well as other, small-scale, events.
The Distributed Training Operations Center plans, builds, and manages  small-scale events to meet the learning objectives of its customers,  mainly Air Combat Command, Air National Guard, and Air Force Reserve  Command. These events are short-term, typically lasting 90 minutes.  During 2011, the Distributed Training Operations Center conducted over  4,000 events that trained more than 9,500 personnel, of which at least 60  percent were active-duty personnel.
The two overseas virtual training centers provide different capabilities for  the commands they support. The Warrior Preparation Center supports  training for joint, coalition, and partner-nation forces in the European and  African theaters. In addition, the Warrior Preparation Center supports an  Air-to-Ground Operations School and three detachments that provide  multinational training opportunities. The Korean Air Simulation Center  operates constructive simulations that support the air operations in Korea  during U.S. Forces Korea operational-level exercises and supports  selected exercises for U.S. Forces Japan.
To train its units and personnel, the Air Force conducts distributed  mission operations using several different internal Air Force and  Department of Defense (DOD) information networks. Some of these key  networks, along with their managing organizations, are shown in table 1.
These networks differ according to such factors as security restrictions,  bandwidth capacity, data protocols, and support services.
In May 2010, the Secretary of Defense directed DOD to undertake a  department-wide efficiency initiative to reduce excess overhead costs and  to reinvest the resulting savings in sustaining force structure and  modernization. The Air Force identified a number of areas to improve its  efficiency, including an initiative, beginning in fiscal year 2012, to  decrease training costs by reducing its live flying hour program for its  legacy fighter and bomber aircraft by 5 percent and its Air Force Reserve  Command F-16 flying hour program by 10 percent.initiative, the Air Force expects to save a total of $1.7 billion from fiscal  years 2012 to 2016. The Air Force estimated savings of about $268  million for fiscal year 2012. In discussing the initiative, the Air Force  stated that it expected to offset any effect on readiness caused by a  reduction in live flying hours by increasing its use of simulators.
section_title: Air Force Major Commands Determine Their Mix of Live and Virtual Training Based on Various Factors
paragraphs: Currently, the three lead Air Force major commands—Air Mobility  Command, Air Force Special Operations Command, and Air Combat  Command—have similar processes to determine the mix of live and  virtual training, but the mix of training differs across the major commands,  and among aircraft within the commands. Air Combat Command is  responsible for fighters, bombers, and attack aircraft; Air Mobility  Command is responsible for transport and tanker aircraft; and Air Force  Special Operations Command is responsible for special-operations  aircraft. At each command, training-requirement review boards composed  of subject-matter experts meet to consider broad sets of training issues  and evaluate training requirements for specific aircraft. The boards  consider factors such as specific combatant command mission  requirements and the capabilities of simulators and networks that have  already been fielded, and determine which training requirements can be  completed in a virtual environment and which need to be completed in a  live environment. The results of their reviews are reflected in updated  training guidance for each type of aircraft. In addition, each of the  commands also establishes requirements to improve, acquire, or upgrade  training devices to meet mission tasks.
While all three lead major commands rely on both live and virtual training  to meet aircrews’ training requirements, the mix is different for each major  command, as discussed below.
subsections: section_title: Air Mobility Command
paragraphs: For each aircraft type, Air Mobility Command issues a requirement  document that specifies the number of times each task or “event” must be  completed for a pilot or aircrew to be certified as mission ready. The  document also specifies the percentage of events that can be completed  in a simulator. For example, the C-130 requirement document specifies  that 50 percent of assault landings may be completed in a simulator and  100 percent of instrument approaches in a simulator. Although live and  virtual training requirements vary by aircraft, according to Air Mobility  Command officials, approximately 50 percent of aircrew training is  conducted in simulators, including all training related to takeoffs, landings,  and instrument approaches.however, training for some special qualifications such as aerial refueling,  formation flying, airdrops, and assault landings must periodically be  conducted live in the actual aircraft. For example, for aerial refueling,  currently, there are differences between what the fighter pilots see in their  simulators and what air refueling crews see in their simulators. Because  the simulators are currently not able to accurately replicate the aerial  refueling environment, simulated training cannot yet replace live training.  In developing its virtual training program, Air Mobility Command worked  with the Federal Aviation Administration to leverage civilian standards,  which require simulators to respond like the actual aircraft in order to be  certified for training. Air Mobility Command is currently developing a  networked distributed training center that would enable more virtual  training with combat air forces and coalition partners.
section_title: Air Force Special Operations Command
paragraphs: As Air Mobility Command does, for each aircraft type Air Force Special  Operations Command issues a requirement document that specifies the  number of times each task or “event” must be completed for a pilot or  aircrew to be certified as mission ready and the percentage of events that  can be completed in a simulator. Air Force Special Operations Command  officials stated that the command’s goal is to accomplish up to 50 percent  of its aircrew training in simulators depending upon the aircraft. For  example, aircrew training requirements for the AC-130U, a close air  support aircraft, allows aircrew to accomplish 50 percent of their mission  tasks in a simulator. Air Force Special Operations Command based its  simulator certification program on the standards and metrics used by the  Federal Aviation Administration and Air Mobility Command. Air Force  Special Operations Command officials stated that simulators provide  training that might not be available in the live environment, such as  training for specific locations or adverse weather conditions. Air Force  Special Operations Command has a stated goal to perform all  qualification and continuation training events in the simulator, while  increasing both live and simulator mission rehearsal training.
section_title: Air Combat Command
paragraphs: Air Combat Command also issues a requirement document for each type  of aircraft on an annual basis. Virtual training requirements vary by  aircraft, with large aircraft such as bombers generally able to satisfy more  of their training requirements in simulators than fighters. Beginning in  fiscal year 2012, Air Combat Command’s training-requirement review  board revised each aircraft’s training requirements and specified that  approximately 25 percent of training requirements were to be met using  virtual training, while the rest of the requirements were to be met using  live training. Prior to this, Air Combat Command training guidance  specified that virtual training was to be used as a supplement to live  training, but it did not set a goal or specific percentage requirement for  virtual training. The virtual training that had been done in those years  included emergency procedures, instrumentation training, and tactical  training rather than mission training. With the availability of more- advanced full mission-training simulators, aircrews are now able to train  beyond these basic tasks to more-advanced air-to-air and air-to-ground  combat missions, like suppression of enemy air defenses.
According to Air Combat Command officials, the combat air forces face  certain challenges that prevent them from conducting the same level of  virtual training as forces from the other major commands. Some  challenges arise due to differences between unit and simulator locations,  difficulties coordinating distributed training events, and a lack of simulator  fidelity. For example, officials stated that there are very few simulators  collocated with Reserve component units, which means valuable reserve  component training time can be lost travelling to and from the simulators.  Officials also noted that the software for some aircraft simulators is two or  three versions behind the software in the actual aircraft, which could in  some cases, affect the performance of aircrews in the actual aircraft. In  addition, fighter simulators cannot replicate the extreme physical effects  of air combat maneuvers that fighter pilots experience in the actual  aircraft. Fighter pilots we interviewed stated that unlike flying other aircraft  such as bombers and transports, fighter pilots must effectively make  decisions while conducting their missions in a hostile environment and  maneuvering the aircraft through high-speed and high-gravity maneuvers  that put stress on the human body. We note that the Navy faces similar  challenges in conducting virtual training for its fighter aircrews. For  example, the crews of the Navy’s F/A-18E/F currently conduct 18 percent  of their training through virtual training and plan to increase this to 32  percent by 2020.
section_title: The Air Force Has Not Established an Overarching Organizational Framework to Guide, Oversee, and Integrate Virtual Training Efforts
paragraphs: The Air Force has recently taken steps to increase management attention  over its virtual training efforts, but its approach to virtual training currently  lacks (1) a designated organization with accountability and authority for  achieving results and (2) an overarching strategy—key elements of an  organizational framework that we have found to be critical for successful  transformations in both public and private organizations. In the absence  of a framework to structure and guide its virtual training efforts, the Air  Force will continue to face challenges in integrating its virtual capabilities  and cannot be certain that its efforts align with strategic goals or know  whether critical gaps or duplication of efforts exist.
subsections: section_title: The Air Force Has Increased Management Focus on Virtual Training Efforts, but Oversight Remains Fragmented
paragraphs: According to Air Force leadership, distributed mission operations are the  cornerstone of the Air Force training transformation. Additionally, in the  Strategic Plan for the Next Generation of Training, DOD has emphasized  comprehensive training that integrates service and joint capabilities. The  Air Force has increased management attention on virtual training efforts  by reorganizing and creating new headquarters offices and establishing  working groups, but oversight remains fragmented. For instance, in  February 2011, the Air Force Agency for Modeling and Simulation was  realigned under Headquarters Air Force Director of Operations (A3O) to  serve as the execution arm for integrating and implementing virtual  capabilities, resources, and policy. Also, in August 2011, the Air Force  established the Headquarters Air Force Director of Operations– Operational Training (A3O-CL) office to provide leadership and support to  distributed mission operations users across the Air Force. Additionally, in  February 2012, Headquarters Air Force established a working group,  composed of subject-matter experts from the Distributed Training Centers  and the major commands, to address operational challenges within the  virtual training programs. Issues unable to be resolved in this forum are  elevated to higher-level working groups including the Headquarters  Operations Conference, the Modeling and Simulation Steering  Committee, and finally the Air Force Modeling and Simulation General  Officer’s Steering Group.
These organizations and working groups have increased management  focus on virtual training efforts, but the Air Force has not designated an  organization with accountability and oversight authority necessary to  integrate all its virtual training efforts, including developing and acquiring  interoperable virtual training systems and establishing and enforcing  authoritative standards for simulators, constructive elements, and  databases. Rather, oversight of standards development, acquisition,  sustainment, and integration of training systems is fragmented among  various Air Force organizations, as shown in table 2.
In the absence of an organization to guide virtual training efforts, the lead  major commands have developed their own standards and acquired and  fielded systems that are not interoperable and often require costly, time- consuming work-arounds to be able to train together in large, complex  virtual training exercises. For example, in conducting its quarterly virtual  flag exercises, the Distributed Mission Operations Center must integrate  simulators and networks that have been developed to different standards.  To integrate all the participants into the exercises, the center must  implement “gateways” that allow dissimilar simulators to translate data;  develop databases to provide a common constructive environment; and  link numerous Air Force and DOD networks that have different security  restrictions, bandwidth limitations, and data transfer protocols. According  to officials, developing and implementing these types of solutions takes  up to 9 months, and involves significant reliance on contractor personnel.  The network configuration must be certified for each virtual flag event and  then it must be disassembled, reconfigured, and recertified for  subsequent training events. While the Air Force has been developing  these work-around solutions to allow interoperability among its older  aircraft simulators, similar solutions will be also be needed for Joint Strike  Fighter simulators as they are fielded and integrated into distributed  mission operations. According to Joint Strike Fighter program office  officials, the programs’ operational requirement document specifies that  the aircraft and simulators must be interoperable with other aircraft and  networks, but interoperability is not scheduled to be achieved until later  blocks in the program’s development. The Air Force’s current Joint Strike  Fighter simulators are stand-alone and not integrated into distributed  mission operations. Like other aircraft simulators, the Joint Strike Fighter  simulators will require complex multilevel security guards and gateways to  allow them to operate with other aircraft simulators in a distributed  mission environment. Joint Strike Fighter program officials stated that  since the types of interoperability challenges that they face are not unique  but are similar to those of other programs there should not be any unique  technical barriers that would prevent solutions to these challenges. Air  Combat Command is also currently assessing the technologies needed to  integrate live and virtual training for the Joint Strike Fighter.
Our prior work has found that designating an integration team, vested  with the necessary authority and resources, is a critical element of  managing the transformation of an organization. We note that the Army  and the Navy have each designated an organization with centralized  oversight over standards development, acquisition, sustainment, and  integration of virtual training systems. Further, the Navy has established  guiding principles and investment priorities to assist decision makers in  selecting the proper simulator solution for specific training requirements  and gaps, and to help avoid interoperability issues. One of the principles  states that simulators intended to interface with other simulators must be  compatible with the Navy Continuous Training Environment network. A  Navy training instruction elaborates on this guiding principle by further  stating, “Interoperability is a key objective for Fleet simulators. Translator  development is expensive and time consuming, translation slows things  down (introduces latency), and translation is never perfect. To reduce the  need for translators to overcome interoperability challenges, Navy  Continuous Training Environment network technical standards have been  adopted Fleet-wide and are mandatory for new simulators that will  integrate into Fleet synthetic training.”
Office of the Under Secretary of Defense, Strategic Plan for the Next Generation of  Training for Department of Defense (Sept. 23, 2010). training environment. However, the Air Force has not yet identified an  organization to perform this oversight. Without a dedicated organization  with responsibility and accountability to integrate virtual training efforts,  the Air Force may continue to face challenges in managing and  integrating its virtual training efforts, including interoperability issues that  lead to diminished training quality, fewer training opportunities due to  lengthy preparation times, and increased costs.
section_title: Individual Air Force Initiatives to Enhance Virtual Training Capabilities Are Not Yet Guided by an Overarching Strategy
paragraphs: The Air Force is currently pursuing a number of individual initiatives to  enhance its virtual training capabilities. Among these initiatives are the  following:
Air Mobility Command is planning to create and staff a Distributed  Training Center at Scott Air Force Base in December 2012. The  training center is initially planned to provide the integration capability  needed for transport aircraft simulators to conduct distributed mission  operations on a daily, consistent basis and will also provide the ability  for these simulators to participate in Distributed Mission Operations  Center events. Air Mobility Command also has plans to expand the  center’s capability to include integration of tanker simulators to train  air refueling virtually.
The Distributed Training Operations Center has increased its  capabilities in response to increased user requirements and mission  requests by adding personnel and expanding event availability.  Additionally, the Distributed Training Operations Center plans to work  with the major commands to establish remote capability sites that  would be networked to the center to increase distributed mission  operations availability across the Air Force. According to Distributed  Training Operations Center officials, remote-capability sites have  already been established for the Air National Guard in South Dakota  and Pacific Air Forces in Alaska.
Air Combat Command is in the process of establishing a Distributed  Training Center at Langley Air Force Base. The training center is  intended to provide a focal point for scheduling of combat air forces  events not involving the Distributed Training Operations Center or  Distributed Mission Operations Center. It is also intended to provide  scenario development, focused on training gaps, desired unit  missions, operational plan missions, and other tactics, techniques,  and procedures. The training center is expected to be fully operational  in September 2012.
Air Force Special Operations Command is in the process of  establishing a Distributed Training Center at Cannon Air Force Base  that will focus on virtual training activities for initial and mission  qualification training and unit-level mission essential tasks.
Our prior work has found that strategic planning is a key element of an  overarching organizational framework. For example, a leading practice  derived from principles established under the Government Performance  and Results Act of 1993 is to improve the management of federal  agencies by developing comprehensive strategies to address  management challenges that threaten their ability to meet long-term  goals. We have previously reported that these types of strategies should  contain results-oriented goals, performance measures, and expectations  with clear linkages to organizational, unit, and individual performance  goals to promote accountability and should also be clearly linked to key  resource decisions. While the Air Force currently has numerous  individual initiatives underway to enhance its virtual training capabilities  and is planning to make additional investments, it has not yet developed  an overall strategy to guide and integrate these efforts. For example, the  Air Force has not outlined overall goals for its virtual training efforts,  resource needs, and investment priorities. In the absence of a strategy,  the Air Force cannot be certain that its individual initiatives are  synchronized and will address its highest priority needs.
Air Force officials stated that they are currently developing a Live, Virtual,  Constructive Flight Plan to serve as their strategy for virtual training. They  told us the Flight Plan will provide direction to the major commands on the  handling of operational issues and will establish an internal structure for  how issues are to be raised and resolved. Officials expect that the Flight  Plan will be completed by July 2012. Officials stated that a separate effort  will be undertaken to develop an acquisition strategy for virtual training  systems. At this point, it is unclear the extent to which these plans will  contain the necessary elements of an overall strategy that the Air Force  can use to manage and integrate its planning and acquisition efforts.
section_title: Air Force Lacks Cost Data for Virtual Training and Did Not Account for Virtual Training Costs When Determining Savings from Its Flying-Hour Efficiency Initiative
paragraphs: In outlining its efficiency initiative related to training, the Air Force  estimated potential cost savings of $268 million for fiscal year 2012, and a  total of $1.7 billion for fiscal years 2012 to 2016 by among other things,  reducing legacy combat Air Force flying hours across the board by 5  percent. The flying-hour efficiency initiative also called for a concurrent  increase in the use of high-fidelity simulators and virtual training to avoid  any effect on aircrews’ mission readiness from the reduction in live flying.  However, in estimating costs, the Air Force included the savings  associated with reductions in live training but not the potential costs  associated with increases in virtual training that were called for to offset  the reduction in live training. On the basis of our prior work, cost savings  estimates should include all significant costs in order to have a  reasonable basis. Additionally, federal internal control standards state  that decision makers need visibility over a program’s financial data to  determine whether the program is meeting the agencies’ goals and  effectively using resources.
Air Force officials told us that the cost savings associated with the flying- hour efficiency initiative were estimated by multiplying the reductions in  live training flying hours for each aircraft by the cost per flying hour for  that aircraft, and then adding the resultant figures for all the aircraft to  determine total savings. For example, according to numbers provided by  the Air Force, the live training cost of 1 F-15E flight hour is approximately  $17,449 and F-15E flying hours were reduced by 1,782 hours. These  amounts were multiplied together to arrive at the Air Force’s projected  total savings of approximately $31,094,000 for reductions in F-15E flight  hours. Similar calculations were made for each of the other aircraft that  had their flight hours reduced and the savings for all the aircraft were  summed.
The Air Force did not consider any potential costs associated with the  increase in virtual training in its estimate of cost savings because it has  not developed a methodology to collect and track information on the cost  of its virtual training program. According to Air Force officials, some  training costs could increase as a result of increases in virtual training.  These costs could include expenses for aircrew to travel to simulator  locations, additional contractor personnel to schedule and operate  simulators, and the purchase of additional simulators to meet increased  demand. Furthermore, according to Air Force officials, identifying virtual  training costs is challenging because funds to support virtual training and  distributed mission operations are currently dispersed across multiple  program elements. For example, our analysis identified a portion of  virtual training funding, specifically distributed mission operations funding,  in a program element titled “Human Effectiveness Applied Research.” In  another case, distributed mission operations funding was part of a  program element titled “International Activities,” under an “Armaments  Cooperation” subcategory that also included funding for alternative  energy among other things. In 2011, the Air Force conducted a onetime  study in an attempt to identify the full cost of its virtual efforts. It found  that the total investment in virtual capabilities for fiscal year 2012 was at  least $1.9 billion. Of that, operational training support accounted for  approximately 50 percent of the annual investment, including the largest  identified expenditure of $182.3 million for combat air forces distributed  mission operations. However, the study noted that its efforts may not  have identified all program elements associated with virtual training and  therefore further steps would be needed to capture the full value of the Air  Force’s virtual training investment. As of May 2012, the Air Force had not  taken any additional steps to develop a methodology for identifying virtual  training costs. Without a means to collect or calculate its virtual training  costs, the Air Force lacks the information it needs to make informed  investment decisions in the future regarding the mix of live and virtual  training. Furthermore, the Air Force will be unable to determine the  potential costs associated with its flying-hour efficiency initiative.
section_title: Conclusions
paragraphs: In an effort to achieve greater efficiencies in its training program while  maintaining mission readiness, the Air Force has taken various steps to  emphasize and increase the use of virtual training. Among other things,  the Air Force has implemented various initiatives and established  organizations intended to enhance its virtual training capabilities.  However, none of these organizations have the authority necessary to  ensure the integration of the Air Force’s virtual training efforts, and  oversight remains fragmented. Further, the Air Force lacks an  overarching organizational framework to guide its current virtual training  efforts and the additional investments it plans to make. In the absence of  such a framework, the Air Force faces challenges in managing its current  inventory of virtual training systems and has experienced delays and  costs that stem from the lack of interoperability among its simulators and  networks, resulting in workarounds that are required to compensate for  these limitations. An overarching management approach, including a  single entity responsible for coordinating and integrating all virtual training  efforts, as well as a comprehensive strategy that aligns individual efforts  with goals and investment priorities, will not be enough if decision makers  lack visibility over the potential costs of virtual training—especially as they  consider future changes to the mix of live and virtual training. Until the Air  Force has a methodology to consistently collect and track its virtual  training costs and a management framework to coordinate its efforts, it  will continue to face challenges to planning and conducting its virtual  training and informing its future investment decisions.
section_title: Recommendations for Executive Action
paragraphs: To develop a fully integrated management approach to guide virtual  training efforts and investments, we recommend the Secretary of Defense  direct the Secretary of the Air Force to designate an entity that is responsible and accountable for integrating  all of the Air Force’s virtual training efforts, including the development  and enforcement of interoperability standards across virtual training  systems, and investment planning; and develop an overarching strategy to align goals and funding for virtual  training efforts across all Air Force major commands. This strategy  should at a minimum contain elements such as results-oriented goals,  performance measures, and a determination of resources needed to  achieve stated goals. In addition, this strategy should show clear  linkages between existing and planned initiatives and goals.
To improve decision makers’ visibility over the costs related to virtual  training, we recommend that the Secretary of Defense direct the  Secretary of the Air Force to develop a methodology for collecting and  tracking cost data for virtual training and use this cost data to help inform  future decisions regarding the mix of live and virtual training.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD stated that it concurred  with all of our recommendations. In response to our recommendation to  designate an entity that is responsible and accountable for integrating all  of the Air Force’s virtual training efforts, including the development and  enforcement of interoperability standards across virtual training systems,  and investment planning, DOD stated that the Air Force has taken initial  steps to designate its Headquarters, Air Force office, AF/A3/5  (Operations, Plans, and Requirements) as the single entity responsible  for integrating the Air Force’s virtual training efforts.
In response to our recommendation to develop an overarching strategy to  align goals and funding for virtual training efforts across all Air Force  major commands, DOD stated that the Air Force is developing an  overarching strategy and policy to provide a fully integrated management  approach to guide its Live Virtual Constructive-Operational Training  efforts and investments. It further stated that operational level guidance  will be provided in Air Force Instruction 36-2251, Management of Air  Force Training Systems and that investment guidance to link virtual  training to “Readiness” was provided in the Air Force’s Fiscal Year 2014  Annual Planning and Programming Guidance and Program Objective  Memoranda Preparation Instructions.
In response to our recommendation to develop a methodology for  collecting and tracking cost data for virtual training and use this cost data  to help inform future decisions regarding the mix of live and virtual  training, DOD stated that the Air Force is taking actions to improve  visibility related to virtual training to inform decisions regarding the mix of  live and virtual training. It also stated that the Air Force Instruction  regarding Management of Air Force Training Systems will provide major  commands with clear guidance to employ consistent methods to collect  and measure virtual training systems data. DOD said the Aviation  Resource Management System will be enhanced to provide the capability  to capture projected and executed aircraft virtual training and cost data to  provide better oversight and management of virtual training funding.  Finally, DOD stated that the Air Force is developing a standard  methodology of accounting and tracking the programming and execution  of program funds through improved visibility into cost categories  associated with Live Virtual Constructive-Operational Training. DOD’s  comments are included in their entirety in appendix II. DOD also provided  a number of technical and clarifying comments, which we have  incorporated where appropriate.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Defense, the Secretary of the Air Force, and  the Under Secretary of Defense for Personnel and Readiness. In addition,  this report will be available at no charge on our website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To address our objectives, we met with officials from the Office of the  Secretary of Defense, Joint Staff, Office of the Secretary of the Air Force,  Headquarters Air Force, and several Air Force major commands. Our  review focused primarily on virtual training systems for manned aircraft  from combat air forces, mobility air forces, and special operations forces.  Excluded from this review were virtual training programs for unmanned  aircraft, space, combat support, and combat service support systems. To  determine how the Air Force determines the mix of live and virtual  training, we obtained and analyzed training requirement instructions for  combat, mobility, and special operations aircraft from each of the three  lead major commands—Air Mobility Command, Air Combat Command,  and Air Force Special Operations Command. We provided a  questionnaire and received written responses from the major commands  on the mix of live and virtual training and the benefits, limitations, and  challenges of virtual training. We interviewed officials from Air Combat  Command, Air Mobility Command, Air Force Special Operations  Command, Air National Guard, Air Force Reserve Command, U.S. Air  Forces Europe, and Pacific Air Forces.
To determine the extent to which the Air Force has developed an  overarching framework to guide, oversee, and integrate its virtual training  efforts, we analyzed Air Force studies on virtual training technologies and  capabilities. We reviewed relevant Department of Defense (DOD) and  Navy training guidance. We also reviewed our ongoing work related to  Navy virtual training. We interviewed officials from the Office of the  Secretary of the Air Force, Headquarters Air Force, the Joint Staff, the  Department of the Navy, Air Force Major Commands, the four primary  centers that facilitate distributed mission operations, and joint training  officials. We visited Langley Air Force Base, Virginia, to observe F-15 and  F-22 simulator operations at the Mission Training Center. We also visited  the Distributed Mission Operations Center at Kirtland Air Force Base,  New Mexico, to observe a distributed training event and the center’s  capabilities.
To determine the extent to which the Air Force considered costs related  to virtual training in estimating potential savings from its training efficiency  initiative, we obtained and analyzed the Air Force efficiency calculation  and compared it with cost saving estimating best practices. We analyzed  Air Force budget program elements related to virtual training and  distributed mission operations. We also analyzed an Air Force study on  the cost associated with Air Force modeling and simulation efforts. We  interviewed officials from the Air Force Defense Contracting Management  Organization and Headquarters Air Force budget operations office, as  well as officials from the major commands listed above.
We conducted this performance audit from August 2011 to July 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Michael J. Ferren, Assistant  Director; Russell M. Bryan; Paige A. Muegenburg; Mathew Sakrekoff; and  Erik S. Wilkins-McKee made key contributions to this report.