Army and Marine Corps Training: Better Performance and Cost Data Needed to More Fully Assess Simulation-Based Efforts
Published: 2013-08-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Army and Marine Corps use live and simulation-based training to meet training goals and objectives. Service officials have noted benefits from the use of simulation-based training--both in terms of training effectiveness and in cost savings or cost avoidance. A House report accompanying the bill for the National Defense Authorization Act for 2012 mandated GAO to review the status of the military services' training programs. This report follows GAO's reports on the Navy and Air Force, and assesses (1) changes in the Army's and Marine Corps' use of simulation-based training, including efforts to integrate live and simulation-based training capabilities; and (2) the factors the Army and Marine Corps consider in determining whether to use live or simulation-based training, including the extent to which they consider performance and cost information. GAO focused on a broad cross-section of occupations (e.g., aviation, armor, artillery), and analyzed service training strategies and other documents; and conducted six site visits and interviewed service officials involved with training and training development for the selected occupations.
section_title: What GAO Found
paragraphs: Over the past several decades, the Army and Marine Corps have increased their use of simulation-based training--simulators and computer-based simulations. Historically, the aviation communities in both services have used simulators to train servicemembers in tasks such as takeoffs, and emergency procedures that could not be taught safely live. In contrast, the services' ground communities used limited simulations prior to 2000. However, advances in technology, and emerging conditions in Iraq and Afghanistan have led to increased use of simulation-based training in the ground forces. For example, in response to increases in vehicle rollovers, both services began using simulators to train servicemembers to safely evacuate vehicles. The services are also collaborating in the development of some simulation-based training devices. For instance, according to Marine Corps officials, the service reused 87 percent of the Army's Homestation Instrumentation Training System's components in its own training system, achieving about $11 million in cost avoidance and saving an estimated 7 years in fielding time. The services are also taking steps to better integrate live and simulation-based training, developing technical capabilities to connect previously incompatible simulation-based training devices. The Army's capability is now being fielded, and the Marine Corps' is in the initial development phase.
The Army and Marine Corps consider various factors in determining whether to use live or simulation-based training, but lack key performance and cost information that would enhance their ability to determine the optimal mix of training and prioritize related investments. As the services identify which requirements can be met with either live or simulation-based training or both, they consider factors such as safety and training mission. Also, they have cited numerous benefits of simulation-based training, such as improving servicemember performance in live training events, and reducing operating costs. Both services rely on subject matter experts, who develop their training programs, and after action reports from deployments and training exercises for information on how servicemembers may have benefited from simulation-based training. However, neither service has established outcome metrics to assist them in more precisely measuring the impact of using simulation-based devices to improve performance or proficiency. Leading management practices recognize that performance metrics can help agencies determine the contributions that training makes to improve results. Army and Marine Corps officials also generally consider simulation-based training to be less costly than live training and analyze some data, such as life cycle costs, when considering options to acquire a particular simulation-based training device. However, once simulation-based training devices are fielded, the services neither reevaluate cost information as they determine the mix of training nor have a methodology for determining the costs associated with simulation-based training. Federal internal control standards state that decision makers need visibility over a program's financial data to determine whether the program is meeting the agencies' goals and effectively using resources. Without better performance and cost data, the services lack the information they need to make more fully informed decisions in the future regarding the optimal mix of training and how best to target investments for simulation-based training capabilities.
section_title: What GAO Recommends
paragraphs: GAO recommends that the services develop metrics, and a methodology to compare live and simulation-based training costs. DOD partially concurred, but noted that it captures all relevant costs needed for decision making. GAO continues to believe the services may not be considering some important simulation-based training costs and a specific methodology is needed to more fully identify the universe of costs needed for comparison purposes.

section_title: Letter
paragraphs: For more than a decade, the Army and Marine Corps have focused their  training on counterinsurgency and stability operations in order to meet  challenges abroad. With the drawdown from Iraq now complete and the  U.S. decreasing its military presence in Afghanistan, more forces are  available for training and the services are transitioning to focus their  training to once again emphasize a broader range of missions. At the  same time, both services face constraints that can affect live training,  such as limitations in maneuver space and access to ranges. As a result,  they must find ways to maximize efficiency and cost-effectiveness in the  use of their training resources. Both services have concluded that  simulation-based training can contribute to these aims and serve as a  complement to live training.
A report of the Committee on Armed Services, House of Representatives,  which accompanied a bill for the National Defense Authorization Act for  2012, mandated that we review the status of the military services’ training  programs and report the results to the House and Senate Armed Services  Committees. For this review we assessed (1) changes in the Army’s and  Marine Corps’ use of simulation-based training devices, including efforts  to integrate live and simulation-based training capabilities; and (2) the  factors the Army and Marine Corps consider in determining whether to  use live or simulation-based training, including the extent to which they  consider performance and cost information.
This is the third report we have produced to assess the services’ mix of  live and simulation-based training. In June 2012, we reported that the  Navy had increased its emphasis on simulation-based training over the  last decade, and had developed an overarching strategy that provided 12  investment priorities for simulation-based training. reported that the Air Force’s approach to managing its virtual training  efforts lacked (1) a designated organization with accountability and  authority for achieving results and (2) an overarching strategy —key  elements of an organizational framework —and, (3) a methodology for  determining the costs of virtual training. Accordingly, we recommended  that the Air Force designate an entity to integrate its virtual training  efforts, develop a strategy to align virtual training initiatives and goals,  and develop a methodology to collect virtual training cost data. DOD  concurred with our recommendations and identified actions that the Air  Force planned to take.
GAO, Navy Training: Observations on the Navy’s Use of Live and Simulated Training,  GAO-12-725R (Washington, D.C.: June 29, 2012). artillery occupations. For the Army we also included infantry, and for the  Marine Corps we also included amphibious assault vehicles and motor  transport occupations.
To determine how the Army’s and Marine Corps’ use of simulation-based  training devices has changed, including efforts to integrate live and  simulation-based training capabilities, we analyzed service documents  that provided information on the historical use of simulation-based training  devices. We also interviewed officials representing the selected  occupations and, among others, the services’ training commands and  materiel development organizations to discuss how simulators and  simulations were and are currently being used. We evaluated guidance  and documentation on the services’ processes for coordinating their  development of simulation-based training capabilities and technologies. In  addition, we reviewed documents on the development of technical  capabilities to connect simulation-based training devices. We also  interviewed, and reviewed documents from, officials at Fort Hood,  Texas—the first installation to field and use the Army’s new technical  capability for connecting live and simulation-based training—to discuss  user assessments.
To ascertain the factors the Army and Marine Corps consider in  determining whether to use live or simulation-based training, including the  extent to which they consider performance and cost information, we  reviewed DOD and service policies and guidance related to developing  and conducting training. We also reviewed documentation on simulator  usage data and met with officials from the services’ training commands to  discuss how this data was being used, and how the services determine  the relationship between simulator usage and performance. In addition,  we met with officials from Department of the Army and both services’  training commands to determine how costs are considered and compared  when developing the different mixes of live and simulation-based training.  We reviewed training documents to determine how the services identified  the mix of training for institutional and home station training. In addition,  we reviewed leading management practices, the federal internal control  standards, and our past reviews on the Navy’s and Air Force’s use of live  and virtual training, and federal cost-estimating and budgeting guidance.
We conducted this performance audit from June 2012 to August 2013, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. For additional information on  our scope and methodology, see appendix I.
subsections: section_title: Background
paragraphs: All Army and Marine Corps training —whether for individuals or units —is  task-based, and each task has an associated set of conditions and  standards. For example, the conditions may specify daytime or nighttime  training, and the standard, a measure of a unit’s or individual’s proficiency  in performing a task, could be to fire a weapon to become familiar with  how it operates, or to put a specific number of rounds on target.
Active and reserve component servicemembers receive both institutional  and home station training during the course of their career. Institutional  training, which includes initial military training, subsequent professional  military education, and leadership training, takes place at schoolhouses,  i.e., Army centers of excellence and Marine Corps formal learning centers  (which include recruit depots). Schoolhouses have designated training  specialties, such as infantry, aviation, and artillery. For some occupations,  the Army and Marine Corps train together at the same location. For  example, both services train their armor and artillery occupations at Fort  Benning, GA and Fort Sill, OK, respectively. During this training,  instructors must closely follow a prescribed program of instruction so  servicemembers can develop, refine, and improve individual skills to  prescribed standards. The goal of institutional training is to ensure a  common base of training and capability that can serve as a foundation for  unit training.
Training at home station builds on the individual skills developed during  institutional training. It begins with individual and small unit training and  builds to large scale culminating training events that are designed to  certify units for deployment. Along the way, training complexity is  increased incrementally by increasing the: task conditions (e.g., adding  nighttime training); training tempo; number of tasks; or, number of  personnel.
Like type units, e.g., all infantry platoons, all Apache helicopter  squadrons, or all Stryker brigade combat teams, train on many of the  same tasks. However, unit commanders are ultimately responsible for  their units’ training, and a variety of factors can lead commanders to  adopt different approaches to training. For example, when units are  scheduled to deploy, commanders may adjust their training based on  their units’ assigned missions or deployment locations.
section_title: Army and Marine Corps Have Increased the Use of Simulation- Based Training and Taken Steps to Collaborate on Development Efforts
paragraphs: Over the last several decades, the Army and Marine Corps have  incorporated simulators and simulations more broadly into training and  are collaborating in the development of these devices. The two services  are also taking steps intended to increase the interoperability of  simulators and simulations and support training across live and  simulation-based training environments.
subsections: section_title: The Services’ Use of Simulation-Based Training Devices Has Increased and They Are Collaborating in Device Development
paragraphs: The Army’s and Marine Corps’ use of simulation-based training devices  has increased over time, and the services have collaborated in  developing some devices. Simulation-based training devices were first  incorporated into training for aircraft and later incorporated into the  ground communities. Both services’ aviation communities have used  simulators for more than half a century. In addition to training tasks that  could not be trained in an actual aircraft, such as emergency procedures,  the services currently use flight simulators to train new pilots on tasks  such as take-offs and landings, and to provide refresher training to more  experienced pilots.
The services’ ground communities did not begin using simulators and  simulations until later. Specifically, until the 1980s, training in the ground  communities was primarily live training. Then, to enhance live, force on  force training, the Army began using a laser training device that simulated  weapons fire. In addition, both services began using simulations for the  purpose of training higher-level commanders and their staffs in command  and control and decision making.
Further advances in technology resulted in the acquisition of simulators  and simulations with additional capabilities designed to help  servicemembers and units acquire and refine skills through more  concentrated and repetitive training. For example, during the 1990s, the  Army introduced more advanced trainers for its ground and aviation  forces. In addition, the Marine Corps began using devices that allowed  individual marines to conduct training in basic and advanced  marksmanship, shoot/no-shoot judgment, and weapons employment  tactics. More recently, during operations in Iraq and Afghanistan, both  services introduced a number of new simulators and simulations to  prepare servicemembers for conditions on the ground and emerging  threats. For example, to provide initial and sustainment driver training, the  Army and Marine Corps began using simulators that can be reconfigured  to replicate a variety of vehicles. In addition, in response to an increase in  vehicle rollovers, both services began using egress trainers to train  servicemembers to safely evacuate their vehicles.
As the Army and Marine Corps have continued to expand their use of  simulation-based training, they have collaborated on the development of  some simulation-based training devices. For example, the Army uses the  Homestation Instrumentation Training System to support collective  maneuver training for platoon through battalion units. In developing a  similar training system— the Marine Corps Instrumentation Training  System —the service determined that it could reuse 87 percent of the  components in the Army’s system. As a result, officials told us, the  Marine Corps achieved approximately $11 million in cost avoidance and  fielded the system in 2 years instead of the projected 9 years. In turn, the  Marine Corps developed enhancements for its own training system that  the Army has incorporated into its fielded version of the Homestation  Instrumentation Training System. Further, the Army and Marine Corps  determined that they had a similar need for terrain maps of Iraq and  Afghanistan in their simulators. Rather than each service acquiring their  own, the Army and Marine Corps share the same version of Virtual  Battlespace 2, a gaming capability, resulting in shared development costs  and content. In addition, the Army’s and Marine Corps’ training materiel  developers, who are collocated in Orlando, Florida, have established  memorandums of understanding intended to promote coordination and  encourage maximum reusability of existing devices. Table 1 provides  examples of simulators and simulations currently being used in the Army  and Marine Corps aviation and ground communities.
As the Army and Marine Corps continue integrating simulation-based  devices into their training, they are taking steps intended to increase  interoperability and support training across live and simulation-based  training environments. The Army’s live, virtual, and constructive  capabilities were largely incompatible and operators at individual  installations had to develop technical workarounds to connect these  capabilities in a single realistic home station training exercise. According  to Army officials, these workarounds are temporary and require significant  time and manpower to prepare for each exercise, which limits the  availability of simulator operators to support other training needs on an  installation. In an effort to overcome the need for temporary solutions and  more fully integrate live, virtual, and constructive training, the Army began  developing the Live, Virtual, Constructive Integrating Architecture (LVC- IA) in 2005. The LVC-IA is a technical capability that provides common  protocols, standards, and interfaces to standardize existing incompatible  devices across installations and facilitate interoperability without  workarounds. According to the Army, the integrated training  environment—facilitated by the LVC-IA —will support unit live and  simulation-based training at the brigade level and below; expand the  training area and mitigate constraints imposed by limited maneuver space  and environmental restrictions; and allow units to replicate the  complexities of the operating environment. Since September 2012, the  Army has fielded the LVC-IA capability at four installations and expects to  field it to a total of 18 Army installations by the end of fiscal year 2017.  Currently, the LVC-IA connects three simulators and simulations,  although service officials told us that they plan to incorporate additional  devices, including those of other services, when they field future versions  of the LVC-IA.
To facilitate the integration of live and simulation-based training devices in  the Marine Corps, the service is currently in the conceptual phase of  development for its Live, Virtual and Constructive Training Environment.  The Marine Corps’ intent is to eventually combine any of the three training  environments (live, virtual and constructive) to create a common  environment by which units can seamlessly interact as though they are  physically located together. After assessing the Army’s LVC-IA, the  Marine Corps concluded that the capabilities identified by the Army were  beneficial to the Marine Corps; however, it did not address all of the  capabilities needed to meet the unique training requirements of the  Marine Corps, such as the need to train across environments while  embarked aboard naval vessels. According to Marine Corps officials, they  are monitoring the Army’s LVC-IA initiative to leverage applicable  technology and lessons learned.
section_title: The Army and Marine Corps Consider Various Factors in Determining the Mix of Live and Simulation-Based Training, But Lack Key Information to Assess the Impact and Cost of Simulation- Based Training
paragraphs: The Army and Marine Corps consider various factors —such as safety,  and training objective or mission—in determining whether to use live or  simulation-based training to meet training requirements. However, the  services do not have information on performance and cost that would  assist them in assessing and comparing the benefits of simulation-based  training as they seek to optimize the mix of training to meet requirements  and prioritize related investments. For example, regarding home station  training, they collect some information on usage of simulation-based  training devices, and the Army’s training materiel developer is conducting  a study to determine the benefits of training specific tasks using  simulations. However, neither service has developed overall metrics or  indicators that can be used on a service-wide basis to measure how the  use of simulation-based training devices contributes to improving the  effectiveness of training. In addition, the services have not developed a  methodology to identify the costs associated with using simulation-based  training.
subsections: section_title: The Services Consider Various Factors to Determine When Live and Simulation-Based Training Can or Will Be Used
paragraphs: The Army and Marine Corps identify requirements and develop related  programs of instruction for institutional training, and overarching training  strategies for home station training. In developing the programs of  instruction and overarching training strategies, officials identify which  requirements can be met with live or simulation-based training and which  can be met using a combination of both types of training. Each type of  training has certain advantages. For example, live training allows  servicemembers to become familiar with the feel and use of actual  weapons or equipment, adjust to uncertain conditions that can change  quickly, such as weather, and work together in teams in a more realistic  environment. Simulation-based training allows servicemembers to quickly  replicate a wide variety of conditions that may not exist on a live training  range, such as terrain differences. It also allows individuals and units to  conduct numerous iterations to master a task (an option that may not be  viable in a live training environment); reduces required equipment  maintenance; and provides instant feedback on performance.
To leverage the advantages associated with both types of training and  determine whether to use live or simulation-based training, officials from  the services consider a number of factors, such as: training objective or  mission; safety of servicemembers conducting the training and the safety  of the general public; required training frequency for the task; available  training time; the need to replicate environmental conditions, e.g.,  weather conditions; availability of training ranges, simulators, and  simulations; and realism of existing simulators and simulations, including  their concurrency.guidance, when deciding whether to use simulation-based training, the  primary consideration is improving the quality of training and the state of  readiness; potential cost savings or avoided costs are an important, but  secondary, consideration.
According to Army officials and Marine Corps training  In developing programs of instruction for institutional training, Army and  Marine Corps subject matter experts, who in many cases have prior  service experience, determine which training tasks will be conducted live,  and which will be met using simulation-based training. For example, Army  aviators learning to fly an Apache helicopter must train for about 55 hours  in a simulation-based environment and about 84 hours in a live  environment. In the same way the Marine Corps prescribes about 66  hours of live training and about 272 hours of simulation-based training for  one of its artillery courses.
For home station training, the services provide unit commanders with  various guidance documents to assist them in developing their unit  training plans. However, the services do not prescribe the environment in  which home station training should be conducted. Instead, both services  allow unit commanders to decide whether to use live or simulation-based  training or a mix of both types of training. While unit commanders have  this discretion, available training time, and the availability of ranges,  ammunition, and simulation-based training devices may influence their  decisions concerning their units’ mix of training.
The Army provides guidance to unit commanders through its Combined  Arms Training Strategies (CATS) and the Marine Corps through Training  and Readiness Manuals. The Army’s CATS identify the tasks to be  trained; purpose; outcomes; training audience; and event duration and  frequency. The CATS also identify which requirements should be  conducted through live training and which can be conducted using a  combination of live and simulation-based training. They also identify the  simulators or simulations that could be used. Similarly, the Marine Corps’  Training and Readiness Manuals identify the critical tasks and associated  standards to which the tasks should be trained; prerequisite training  events; the internal and external support necessary to complete an event;  as well as the simulation-based training devices that can be used. In  some instances, the Marine Corps manuals are more specific than the  Army CATS because they contain simulation codes that specify when a  simulator or simulation must, should, or can be used, as well as when no  simulation-based training device is available. However, there are very few  instances where the manuals specify that simulation-based training must  be used. Both services’ training strategies recommend that  servicemembers and units use simulators or simulations to build and  maintain proficiency prior to qualifying for many tasks in a live training  environment.
Like the Army and Marine Corps, the Air Force and Navy recognize that  effective training requires a mix of live and simulation-based training and  some live training events cannot or should not be replaced by a simulator.  As we previously reported, the Navy and Air Force have identified mixes  of live and simulation-based training and in some cases are more  prescriptive with regard to simulation-based training and their investments  in these training technologies. In particular, the Air Force has identified  the percentage of events that can be completed in a simulator and the  Navy’s Overarching Fleet Training Simulator Strategy contains specific  guiding principles for simulator use. For example, one principle states that  training simulators should be used to replace live training to the maximum  extent possible where training effectiveness and operational readiness  are not compromised. Another states that if a skill or talent can be  developed or refined, or if proficiency can effectively and efficiently be  maintained in a simulator, then a simulator should be used. Furthermore,  recognizing the constrained fiscal environment and pressure on defense  accounts, the strategy lists 12 investment priorities, including investing in  simulators and simulations that have the greatest potential to generate  cost savings, and it assigns responsibility for developing a methodology  for tracking return on simulator investments.
section_title: The Army and Marine Corps Cite Benefits of Simulation-Based Training, but Lack Information to Evaluate Impacts on Performance and Cost
paragraphs: Army and Marine Corps training documents and officials from both  services have noted benefits from the use of simulation-based training — both in terms of training effectiveness and in terms of cost savings or cost  avoidance. According to various documents from both services, training in  a simulation-based environment complements necessary live training and  allows the fundamentals to be practiced in a more cost-effective manner.  Further, simulation-based training allows servicemembers to quickly  replicate a wide variety of conditions, receive instant feedback on  performance, and conduct multiple iterations to master a task. In addition,  training with simulation-based devices can improve training efficiency by:  controlling costs by expanding the number of training sites; reducing  training time; improving safety and reducing equipment wear and tear;  reducing or offsetting training ammunition requirements and/or operating  tempo costs; and reducing the need for additional training land. Finally,  training officials from both the Army and the Marine Corps noted the  improved performance of servicemembers in a live training environment  as a result of the increased use of simulation-based training. However,  the services lack information to assist them in better determining how the  use of simulation-based training devices contribute to improved  performance and evaluating the costs of simulation-based training.
While the Army and Marine Corps currently collect data on the usage of  simulators and the Army is conducting a study to gain insight into how  simulation-based training contributes to training, neither service has  established metrics or indicators to assist them in more broadly  measuring the impact of simulation-based training on improving the  performance or proficiency of servicemembers. As the services look at  ways to optimize the use of training resources, such information could be  useful to guide decisions on the optimal mix of live and simulation-based  training during the training development process, and as commanders  exercise discretion in how best to integrate the use of simulators during  home station training. Specifically, we found that the services collect  utilization data to manage the scheduling and distribution of simulation- based training devices across installations to support home station  training. For example, on a monthly basis, both the Army and Marine  Corps collect data on the number of hours simulators are used and  throughput (i.e., the number of soldiers or marines who use a simulator),  and in some cases, the number of virtual rounds fired or miles driven.  Further, in the case of the Army, the services’ training materiel developer  is currently conducting a study to determine the benefits of simulation- based training. Specifically, the Program Executive Office for Simulations,  Training, and Instrumentation, is compiling case studies to demonstrate  the benefits and impacts of training certain tasks using simulations. The  intent of this study is to evaluate how training with simulation-based  devices compares to training without these devices. The Army’s study,  which it expects to release later in 2013, as well as the utilization data  both services are collecting, could be useful in establishing metrics to  help optimize the services’ use of their training resources. Neither service  has taken steps to identify performance metrics and the type of  performance data that would be needed to evaluate how the use of  simulation-based training devices contributes to training effectiveness.  Officials told us they recognize the value of performance metrics, but  given the pace of operations in Iraq and Afghanistan over the past several  years, priority was focused on conducting training and preparing forces to  deploy. Officials further noted that some training tasks are subjective,  making it difficult to develop specific, quantifiable metrics, and that  different devices may require different sets of metrics to reflect how their  use contributes to improved performance. Currently, in the absence of  performance data, the services obtain information on the contribution of  simulation-based training from subject matter experts, who are  responsible for developing training programs of instruction and  overarching strategies, as well as information based on feedback and  after action reports from deployments and training exercises.
As previously noted, the Navy has identified guiding principles for  simulator use, including a principle that notes the need to quantifiably  demonstrate how simulator use contributes to achieving training  objectives. Further, leading management practices recognize, when  designed effectively, performance measures help decision makers (1)  determine the contributions that training makes to improve results, (2)  identify gaps in performance, and (3) determine where to focus resources  to improve results. In particular, incorporating valid measures of  effectiveness, i.e., outcome measures, into training programs would  enable an organization to better ensure that desired changes will occur in  trainees’ skills, knowledge, and abilities. We recognize and have  previously reported that it is difficult to establish performance measures  for outcomes that are not readily observable.to determine how the use of simulation-based training devices contribute  to improved performance, decision makers in the Army and Marine Corps  lack information to make informed decisions about the optimal mix of  training.
GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1  (Washington, D.C.: Nov. 1, 1999).
Force’s use of live and virtual training, for instance, we reported that the  Air Force had estimated it could realize savings in its training program by  reducing live flying hours and taking other steps, such as increasing the  use of virtual training. However, the Air Force did not have a complete  picture of costs related to virtual training. For example, it had excluded  certain costs from its estimates, such as expenses for aircrew travel to  simulator locations, additional contractor personnel to schedule and  operate simulators, and the purchase of additional simulators to meet  increased demand. Therefore, we recommended that the Air Force  develop a methodology to determine the universe of costs and a means  to collect and track data in order to enhance its ability to make future  investment decisions about the mix of live and virtual training. DOD  concurred with our recommendation and noted that the Air Force is  developing a standard methodology of accounting and tracking cost  categories associated with live and simulation-based training.
In general, the Army and Marine Corps collect and assess some costs  associated with the use of simulation-based training devices as part of  their acquisition and budgeting processes. For example, both services  conduct cost benefit analyses—including a review of estimated simulator  and simulation life-cycle costs, such as development costs and costs to  dispose of a device at the end of its life—as they make acquisition  decisions such as whether to develop new simulation-based training  devices. In addition, after the simulator or simulation is acquired and  fielded, the services consider the life-cycle cost to operate and maintain  them as they identify the funding needs to be considered in the budget  development process. However, they do not reevaluate cost information  during their training development process as they determine which  training should be conducted live and which can be conducted using a  combination of live and simulation-based training. Further, at this point in  time, neither service has a methodology for identifying the universe of  costs associated with using simulation-based devices or a means to  collect and track these costs.
According to Army and Marine Corps officials, additional cost information  would be useful in making decisions on the mix of training and related  investment decisions. During the course of our work, some officials cited  examples of specific costs that could be considered and variables that  might need to be taken into account in developing an approach to  comparing simulation-based and live training. For example, some officials  noted that ammunition costs could be considered significant to both types  of training, but that there are some variables to be considered in  comparing these costs. For instance, they noted that the number of virtual  rounds fired in a simulator could be compared to the cost of live  ammunition, but that there are limits to this type of comparison because  units can fire more rounds in a simulator than they would be allocated  during live training. As a result, a one-to-one comparison does not  present a totally accurate picture of the potential costs that are saved or  avoided when conducting simulation-based training rather than live  training. Additionally, they noted that ammunition has a shelf life. If it is  not used within its available shelf life the services can incur costs to  demilitarize the ammunition. Therefore, firing live ammunition that is  approaching the end of its shelf life may actually result in cost avoidance  rather than an additional cost for the services. In addition, they noted that  comparing the costs of fuel and spare parts of an aircraft used in live  training to the cost of technicians and spare parts needed to maintain  simulators would not provide an accurate comparison if the costs of  facilities, utilities, and training personnel to support the simulators were  not included as well. These costs and variables could serve as the  foundation for developing a cost methodology. Without a means to  assess the impact of using simulators on performance and to compare  the costs associated with live training and the use of simulation-based  training devices, decision makers in the Army and Marine Corps lack  information to make fully informed decisions in the future regarding the  optimal mix of training and related investment decisions.
section_title: Conclusions
paragraphs: As the Army and Marine Corps take steps to further integrate the use of  simulation-based training with live training, and collaborate on  development efforts in a fiscally constrained environment, both services  are facing important decisions regarding how to adapt current approaches  to meet training requirements and prioritize related investments, including  those related to the acquisition of simulation-based training capabilities.  While both services have noted benefits from the use of simulation-based  training —in terms of training effectiveness and in cost savings or cost  avoidance, it is important that they have valid performance and cost data  to assist them in evaluating these benefits. We recognize that both  services currently collect various types of information on the use of  simulation devices and consider costs to a certain extent in their  acquisition and budgeting processes. However, taking additional steps to  expand these efforts by establishing performance-oriented metrics and a  methodology to identify the costs associated with simulation-based  training would provide them greater insights into how the use of  simulation-based training contributes to improved performance or  proficiency of servicemembers, and a point of comparison for assessing  the cost implications of using simulation-based or live training. Moreover,  until the Army and Marine Corps take actions to increase their visibility  over the impact of simulation-based training on performance and costs,  they will continue to lack key information that could assist them in  determining how to optimize the mix of live and simulation-based training  in the future and target simulation-based training investments on the  devices that have the greatest potential to improve mission performance.
section_title: Recommendations for Executive Action
paragraphs: To improve decision makers’ abilities to make fully informed decisions  concerning whether training requirements can be met with live and  simulation-based training and determine optimal mixes of live and  simulation-based training, we recommend that the Secretary of Defense  direct the Secretary of the Army and the Commandant of the Marine  Corps to take the following two actions:
Develop outcome-oriented performance metrics that can be used to  assess the impact of simulation-based training on improving the  performance or proficiency of servicemembers and units.
Develop a methodology—to include identifying the costs that should  be included and how these costs should be captured—for comparing  the costs associated with the use of live and simulation-based  training.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD partially concurred with  our recommendations. In response to our recommendation that the  Secretary of Defense direct the Secretary of the Army and the  Commandant of the Marine Corps to develop outcome-oriented  performance metrics that can be used to assess the impact of simulation- based training on improving the performance or proficiency of  servicemembers and units, DOD agreed that an enhancement in  outcome-oriented performance metrics would be helpful in the decision- making process. DOD noted that given the magnitude and scope of  training tasks, varying competencies of the training audience, and ever  changing technology, the problem set contains many independent  variables. DOD said that it will study the problem set, and as appropriate,  develop a construct and implementation plan to include performance  metrics to assess the impact of simulation-based training on improving  the performance or proficiency of servicemembers and units. We  recognize that DOD must consider many independent variables and note  in our report that some training tasks are subjective, making it difficult to  develop specific, quantifiable metrics. However, facing these same types  of challenges, we describe in our report that the Navy has a simulator  strategy that notes the need to quantifiably demonstrate how simulator  use contributes to achieving objectives. Further, as noted in our report,  Army and Marine Corps training tasks already have associated conditions  and standards. These standards could be used as the basis for  developing performance metrics to evaluate differences, if any, between  live and simulation-based training effectiveness. As both services have  noted anecdotal benefits from the use of simulation-based training, we  continue to believe that establishing performance-oriented metrics would  provide the Army and Marine Corps with greater insights into how the use  of simulation-based training contributes to improved performance or  proficiency.
DOD partially concurred with our recommendation that the Secretary of  Defense direct the Secretary of the Army and the Commandant of the  Marine Corps to develop a methodology—to include identifying the costs  that should be included and how these costs should be captured—for  comparing the costs associated with the use of live and simulation-based  training. In its comments, DOD noted that the Army and Marine Corps  capture all relevant costs needed for decision-making during the  Planning, Programming, Budgeting, and Execution (PPBE) process in  procuring simulators/simulation devices. DOD further stated that the  Marine Corps training is based on the Systems Approach to Training,  which includes policy on developing outcome-oriented performance  metrics that are employed in school house lesson plans and home station  training. DOD noted that the combination of PPBE and the Systems  Approach to Training ensures that costs are considered in determining  the mix of live and simulation-based training. DOD further stated that the  Army and Marine Corps concur that a more comprehensive cost analysis  would assist decision making in determining the optimal mix of live and  simulation-based training. DOD did not cite any specific steps that the  services plan to take. In our report, we specifically recognize that the  Army and Marine Corps assess costs associated with simulation based  devices, such as life cycle costs, as they make acquisition decisions and  during their budget development process when they determine funding  needs to operate acquired devices. However, the services do not  reevaluate cost information during their training development process  which is the point at which they are determining the mix of live and  simulated based training. Furthermore, beyond those costs currently  assessed in the budget process, we found examples of additional costs  that could be considered if the services were to perform a cost  comparison analysis between live and simulated based training, such as  facilities, fuel and ammunition. Therefore, to enhance their ability to  optimize the mix of training and better understand related cost  implications, we continue to believe the services need to expand their  current efforts and take specific steps to develop a methodology for  comparing costs associated with the use of live and simulated based  training. DOD’s comments are included in their entirety in appendix II.  DOD also provided technical and clarifying comments, which we have  incorporated where appropriate.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Defense, the Secretary of the Army, and the  Commandant of the Marine Corps. In addition, this report will be available  at no charge on our website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To address our objectives, we met with officials from the Office of the  Secretary of Defense, the Department of the Army Headquarters, Marine  Corps Headquarters, and several Army and Marine Corps commands and  organizations, and visited simulation-based training facilities. Our review  focused on the mix of live and simulation-based training for institutional  and home station training because this training represents a significant  portion of both services’ training. Excluded from this review were live and  simulation-based training at the services’ combat training centers and  during deployment. We selected Army and Marine Corps occupations that  use the largest number of simulation-based training devices and  represent a broad cross-section of how these devices are used by the  services, respectively. For both the Army and Marine Corps, we selected  the aviation, armor, and artillery occupations. In addition, for the Army we  included infantry, and for the Marine Corps we included amphibious  assault vehicles and motor transport occupations. We held discussions  with training officials representing each of these occupations. In addition,  we visited the Army Aviation Center of Excellence in Fort Rucker,  Alabama; the Army Maneuver Center of Excellence at Fort Benning, GA,  where both services train their armor personnel; and Fort Sill, Oklahoma,  where both services train their artillery personnel.
To determine how the Army’s and Marine Corps’ use of simulation-based  training devices has changed since the services first began using  simulators, we reviewed and analyzed service briefings and  documentation that provided information on the historical use of  simulation-based training devices, and the timelines within which  simulators for various occupations became available. In addition, we  interviewed officials from the Department of the Army-Management Office  for Training Simulations; Marine Corps Plans, Policies and Operations;  and the services’ training commands —the Army’s Training and Doctrine  Command and the Marine Corps’ Training and Education Command—to  discuss how simulators and simulations were and are currently being  used. We also interviewed officials representing the selected service  occupations and from the Army Reserve Command, the Army National  Guard, and Marine Corps Forces Reserve, to discuss the mix of live and  simulation-based training, how the use of simulators and simulations has  evolved, and the benefits, limitations, and challenges of simulation-based  training. Additionally, we reviewed our prior reports to gain additional  perspective on how simulators and simulations were used in the past. To  assess the Army’s and Marine Corps’ efforts to better integrate live and  simulation-based training, we evaluated documentation on simulation- based training technologies and capabilities. In addition, we met with  officials from the Army’s and Marine Corps’ training commands and  materiel development organizations—the Program Executive Office for  Simulations, Training, and Instrumentation and the Program Manager for  Training Systems, respectively—to discuss how they coordinate the  development of simulation-based training capabilities and technologies  and ongoing initiatives to further integrate simulation-based training. We  also obtained and reviewed guidance and documentation from both  services related to the development of certain technical capabilities to  connect incompatible simulation-based training devices—the Army’s Live,  Virtual, and Constructive—Integrated Architecture (LVC-IA), and the  Marine Corps’ Live, Virtual and Constructive Training Environment. We  discussed these technical capabilities with officials from the Army  headquarters, and the services’ training commands and materiel  development organizations. In addition, we reviewed Army documents on  the LVC-IA and the summary of the findings and lessons learned from the  initial fielding of the integrated architecture at Fort Hood, Texas. We also  interviewed officials from Fort Hood, including company, battalion, and  brigade-level officials from the 2nd Brigade Combat Team, 1st Cavalry  Division, to discuss their perspectives on the benefits and challenges  related to the initial LVC-IA exercise. Additionally, we reviewed Marine  Corps’ documents, such as its Live, Virtual and Constructive Training  Environment Initial Capabilities Document; and the Training and  Education Modeling and Simulation Master Plan 2010. We also  interviewed Marine Corps training officials to obtain information on the  purpose and current state of its Live, Virtual and Constructive Training  Environment.
To ascertain the factors the Army and Marine Corps consider in  determining whether to use live or simulation-based training, including the  extent to which they consider performance and cost information, we  assessed Army and Marine Corps documentation, such as the 2013 Army  Posture Statement; U.S Army Training Concept 2012-2020; Army  Regulation 350-38, Policies and Management for Training Aids, Devices,  Simulators, and Simulations; the 2012 Army Training Strategy; the  Posture of the United States Marine Corps, 2013 Report to Congress;  and Marine Corps Concepts and Programs 2013. We reviewed  Department of Defense (DOD) and Navy Training guidance. We also  reviewed our previous reports on Air Force and Navy virtual training. To  determine the services’ mixes of live and simulation-based training for  institutional and home station training, we reviewed Army and Marine  Corps policies and guidance related to developing and conducting  training, including the Training and Doctrine Command Regulation 350- 70: Army Learning Policy and Systems; U.S. Army Training and Doctrine  Command Pamphlet 350-70-1, Training Development in Support of the  Operational Domain; and NAVMC 1553.1: the Marine Corps’ Systems  Approach to Training User’s Guide. We interviewed officials from the  services’ training commands, the Army Reserve Command, the Army  National Guard, Marine Corps Forces Reserve, and subject matter  experts for the selected occupations to discuss the mix of training at  institutions and home station for active and reserve component  personnel, how these mixes are developed, and the factors that are  considered. We reviewed programs of instruction to obtain examples of  the mixes of live and simulation-based training prescribed during  institutional training in the selected occupations. To determine the  services’ mix of training at home station we reviewed examples of the  Army Combined Arms Training Strategies, Army proponent’s Home  Station Gated Training Strategies, and the Marine Corps’ Training and  Readiness Manuals. We also interviewed officials from the Army Reserve  and Army National Guard, as well as unit commanders and officials from  the installations that we visited, i.e., Army’s 3rd Brigade, 3rd Infantry  Division at Fort Benning, Georgia; 1st Air Cavalry Brigade, 1st Calvary  Division, at Fort Hood, Texas; 31st Air Defense Artillery Brigade, 75th  Fires Brigade, and 214th Fires Brigade at Fort Sill, Oklahoma; and former  Marine Corps battalion and company commanders and officials from the  Marine Corps Forces Reserve to discuss how the Army’s and Marine  Corps’ overarching training strategies assist commanders in developing a  mix of live and simulation-based training for home station training. In  addition, we interviewed Army and Marine Corps training officials to  identify and determine the types of information collected on the use of  simulators and simulations; how this information was being used; and  how the services determine the relationship between simulator usage and  performance. We also obtained examples of the types of information  being collected by both services, and participated in an online  demonstration of the Army’s central repository for this information. In  addition, we reviewed the Navy’s Overarching Fleet Simulator Strategy  and management practices on performance measures. To determine how  costs are considered and compared when developing live and simulation- based training, we met with officials from Department of the Army and  both services’ training commands. We reviewed the Army’s 3rd edition of  its Cost Benefit Analysis Guide to identify potential DOD criteria as to  what factors should be included when conducting a cost-benefit analysis.  We also reviewed Army and Marine Corps acquisition guidance, which  provides instruction on conducting cost-benefit analyses to acquire  simulators. In addition, we reviewed federal internal control standards, our  past reviews on the Air Force’s use of live and virtual training, and federal  cost-estimating and budgeting guidance. Further, we interviewed training  officials and current and former unit commanders from both services to  further clarify how costs are considered when developing the different  mixes of live and simulation-based training.
In conducting this work, we contacted officials from the organizations  outlined in table 2.
We conducted this performance audit from June 2012 to August 2013, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Sharon Pickup, (202) 512-9619 or pickups@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, key contributors to this report  were Michael Ferren, Assistant Director; Richard Burkard; Kenya Jones;  Jeff Rankin; Michael Silver; Susan Tindall; Erik Wilkins-McKee; and  Richard Winsor.