Army and Marine Corps Training: Better Performance and Cost Data Needed to More Fully Assess Simulation-Based Efforts
Released: 2013-08-22T00:00:00

Letter

Background

All Army and Marine Corps training —whether for individuals or units —is task-based, and each task has an associated set of conditions and standards. For example, the conditions may specify daytime or nighttime training, and the standard, a measure of a unit’s or individual’s proficiency in performing a task, could be to fire a weapon to become familiar with how it operates, or to put a specific number of rounds on target.

Active and reserve component servicemembers receive both institutional and home station training during the course of their career. Institutional training, which includes initial military training, subsequent professional military education, and leadership training, takes place at schoolhouses, i.e., Army centers of excellence and Marine Corps formal learning centers (which include recruit depots). Schoolhouses have designated training specialties, such as infantry, aviation, and artillery. For some occupations, the Army and Marine Corps train together at the same location. For example, both services train their armor and artillery occupations at Fort Benning, GA and Fort Sill, OK, respectively. During this training, instructors must closely follow a prescribed program of instruction so servicemembers can develop, refine, and improve individual skills to prescribed standards. The goal of institutional training is to ensure a common base of training and capability that can serve as a foundation for unit training.

Training at home station builds on the individual skills developed during institutional training. It begins with individual and small unit training and builds to large scale culminating training events that are designed to certify units for deployment. Along the way, training complexity is increased incrementally by increasing the: task conditions (e.g., adding nighttime training); training tempo; number of tasks; or, number of personnel.

Like type units, e.g., all infantry platoons, all Apache helicopter squadrons, or all Stryker brigade combat teams, train on many of the same tasks. However, unit commanders are ultimately responsible for their units’ training, and a variety of factors can lead commanders to adopt different approaches to training. For example, when units are scheduled to deploy, commanders may adjust their training based on their units’ assigned missions or deployment locations.

Army and Marine Corps Have Increased the Use of Simulation- Based Training and Taken Steps to Collaborate on Development Efforts

Over the last several decades, the Army and Marine Corps have incorporated simulators and simulations more broadly into training and are collaborating in the development of these devices. The two services are also taking steps intended to increase the interoperability of simulators and simulations and support training across live and simulation-based training environments.

The Services’ Use of Simulation-Based Training Devices Has Increased and They Are Collaborating in Device Development

The Army’s and Marine Corps’ use of simulation-based training devices has increased over time, and the services have collaborated in developing some devices. Simulation-based training devices were first incorporated into training for aircraft and later incorporated into the ground communities. Both services’ aviation communities have used simulators for more than half a century. In addition to training tasks that could not be trained in an actual aircraft, such as emergency procedures, the services currently use flight simulators to train new pilots on tasks such as take-offs and landings, and to provide refresher training to more experienced pilots.

The services’ ground communities did not begin using simulators and simulations until later. Specifically, until the 1980s, training in the ground communities was primarily live training. Then, to enhance live, force on force training, the Army began using a laser training device that simulated weapons fire. In addition, both services began using simulations for the purpose of training higher-level commanders and their staffs in command and control and decision making.

Further advances in technology resulted in the acquisition of simulators and simulations with additional capabilities designed to help servicemembers and units acquire and refine skills through more concentrated and repetitive training. For example, during the 1990s, the Army introduced more advanced trainers for its ground and aviation forces. In addition, the Marine Corps began using devices that allowed individual marines to conduct training in basic and advanced marksmanship, shoot/no-shoot judgment, and weapons employment tactics. More recently, during operations in Iraq and Afghanistan, both services introduced a number of new simulators and simulations to prepare servicemembers for conditions on the ground and emerging threats. For example, to provide initial and sustainment driver training, the Army and Marine Corps began using simulators that can be reconfigured to replicate a variety of vehicles. In addition, in response to an increase in vehicle rollovers, both services began using egress trainers to train servicemembers to safely evacuate their vehicles.

As the Army and Marine Corps have continued to expand their use of simulation-based training, they have collaborated on the development of some simulation-based training devices. For example, the Army uses the Homestation Instrumentation Training System to support collective maneuver training for platoon through battalion units. In developing a similar training system— the Marine Corps Instrumentation Training System —the service determined that it could reuse 87 percent of the components in the Army’s system. As a result, officials told us, the Marine Corps achieved approximately $11 million in cost avoidance and fielded the system in 2 years instead of the projected 9 years. In turn, the Marine Corps developed enhancements for its own training system that the Army has incorporated into its fielded version of the Homestation Instrumentation Training System. Further, the Army and Marine Corps determined that they had a similar need for terrain maps of Iraq and Afghanistan in their simulators. Rather than each service acquiring their own, the Army and Marine Corps share the same version of Virtual Battlespace 2, a gaming capability, resulting in shared development costs and content. In addition, the Army’s and Marine Corps’ training materiel developers, who are collocated in Orlando, Florida, have established memorandums of understanding intended to promote coordination and encourage maximum reusability of existing devices. Table 1 provides examples of simulators and simulations currently being used in the Army and Marine Corps aviation and ground communities.

As the Army and Marine Corps continue integrating simulation-based devices into their training, they are taking steps intended to increase interoperability and support training across live and simulation-based training environments. The Army’s live, virtual, and constructive capabilities were largely incompatible and operators at individual installations had to develop technical workarounds to connect these capabilities in a single realistic home station training exercise. According to Army officials, these workarounds are temporary and require significant time and manpower to prepare for each exercise, which limits the availability of simulator operators to support other training needs on an installation. In an effort to overcome the need for temporary solutions and more fully integrate live, virtual, and constructive training, the Army began developing the Live, Virtual, Constructive Integrating Architecture (LVC- IA) in 2005. The LVC-IA is a technical capability that provides common protocols, standards, and interfaces to standardize existing incompatible devices across installations and facilitate interoperability without workarounds. According to the Army, the integrated training environment—facilitated by the LVC-IA —will support unit live and simulation-based training at the brigade level and below; expand the training area and mitigate constraints imposed by limited maneuver space and environmental restrictions; and allow units to replicate the complexities of the operating environment. Since September 2012, the Army has fielded the LVC-IA capability at four installations and expects to field it to a total of 18 Army installations by the end of fiscal year 2017. Currently, the LVC-IA connects three simulators and simulations, although service officials told us that they plan to incorporate additional devices, including those of other services, when they field future versions of the LVC-IA.

To facilitate the integration of live and simulation-based training devices in the Marine Corps, the service is currently in the conceptual phase of development for its Live, Virtual and Constructive Training Environment. The Marine Corps’ intent is to eventually combine any of the three training environments (live, virtual and constructive) to create a common environment by which units can seamlessly interact as though they are physically located together. After assessing the Army’s LVC-IA, the Marine Corps concluded that the capabilities identified by the Army were beneficial to the Marine Corps; however, it did not address all of the capabilities needed to meet the unique training requirements of the Marine Corps, such as the need to train across environments while embarked aboard naval vessels. According to Marine Corps officials, they are monitoring the Army’s LVC-IA initiative to leverage applicable technology and lessons learned.

The Army and Marine Corps Consider Various Factors in Determining the Mix of Live and Simulation-Based Training, But Lack Key Information to Assess the Impact and Cost of Simulation- Based Training

The Army and Marine Corps consider various factors —such as safety, and training objective or mission—in determining whether to use live or simulation-based training to meet training requirements. However, the services do not have information on performance and cost that would assist them in assessing and comparing the benefits of simulation-based training as they seek to optimize the mix of training to meet requirements and prioritize related investments. For example, regarding home station training, they collect some information on usage of simulation-based training devices, and the Army’s training materiel developer is conducting a study to determine the benefits of training specific tasks using simulations. However, neither service has developed overall metrics or indicators that can be used on a service-wide basis to measure how the use of simulation-based training devices contributes to improving the effectiveness of training. In addition, the services have not developed a methodology to identify the costs associated with using simulation-based training.

The Services Consider Various Factors to Determine When Live and Simulation-Based Training Can or Will Be Used

The Army and Marine Corps identify requirements and develop related programs of instruction for institutional training, and overarching training strategies for home station training. In developing the programs of instruction and overarching training strategies, officials identify which requirements can be met with live or simulation-based training and which can be met using a combination of both types of training. Each type of training has certain advantages. For example, live training allows servicemembers to become familiar with the feel and use of actual weapons or equipment, adjust to uncertain conditions that can change quickly, such as weather, and work together in teams in a more realistic environment. Simulation-based training allows servicemembers to quickly replicate a wide variety of conditions that may not exist on a live training range, such as terrain differences. It also allows individuals and units to conduct numerous iterations to master a task (an option that may not be viable in a live training environment); reduces required equipment maintenance; and provides instant feedback on performance.

To leverage the advantages associated with both types of training and determine whether to use live or simulation-based training, officials from the services consider a number of factors, such as: training objective or mission; safety of servicemembers conducting the training and the safety of the general public; required training frequency for the task; available training time; the need to replicate environmental conditions, e.g., weather conditions; availability of training ranges, simulators, and simulations; and realism of existing simulators and simulations, including their concurrency.guidance, when deciding whether to use simulation-based training, the primary consideration is improving the quality of training and the state of readiness; potential cost savings or avoided costs are an important, but secondary, consideration.

According to Army officials and Marine Corps training In developing programs of instruction for institutional training, Army and Marine Corps subject matter experts, who in many cases have prior service experience, determine which training tasks will be conducted live, and which will be met using simulation-based training. For example, Army aviators learning to fly an Apache helicopter must train for about 55 hours in a simulation-based environment and about 84 hours in a live environment. In the same way the Marine Corps prescribes about 66 hours of live training and about 272 hours of simulation-based training for one of its artillery courses.

For home station training, the services provide unit commanders with various guidance documents to assist them in developing their unit training plans. However, the services do not prescribe the environment in which home station training should be conducted. Instead, both services allow unit commanders to decide whether to use live or simulation-based training or a mix of both types of training. While unit commanders have this discretion, available training time, and the availability of ranges, ammunition, and simulation-based training devices may influence their decisions concerning their units’ mix of training.

The Army provides guidance to unit commanders through its Combined Arms Training Strategies (CATS) and the Marine Corps through Training and Readiness Manuals. The Army’s CATS identify the tasks to be trained; purpose; outcomes; training audience; and event duration and frequency. The CATS also identify which requirements should be conducted through live training and which can be conducted using a combination of live and simulation-based training. They also identify the simulators or simulations that could be used. Similarly, the Marine Corps’ Training and Readiness Manuals identify the critical tasks and associated standards to which the tasks should be trained; prerequisite training events; the internal and external support necessary to complete an event; as well as the simulation-based training devices that can be used. In some instances, the Marine Corps manuals are more specific than the Army CATS because they contain simulation codes that specify when a simulator or simulation must, should, or can be used, as well as when no simulation-based training device is available. However, there are very few instances where the manuals specify that simulation-based training must be used. Both services’ training strategies recommend that servicemembers and units use simulators or simulations to build and maintain proficiency prior to qualifying for many tasks in a live training environment.

Like the Army and Marine Corps, the Air Force and Navy recognize that effective training requires a mix of live and simulation-based training and some live training events cannot or should not be replaced by a simulator. As we previously reported, the Navy and Air Force have identified mixes of live and simulation-based training and in some cases are more prescriptive with regard to simulation-based training and their investments in these training technologies. In particular, the Air Force has identified the percentage of events that can be completed in a simulator and the Navy’s Overarching Fleet Training Simulator Strategy contains specific guiding principles for simulator use. For example, one principle states that training simulators should be used to replace live training to the maximum extent possible where training effectiveness and operational readiness are not compromised. Another states that if a skill or talent can be developed or refined, or if proficiency can effectively and efficiently be maintained in a simulator, then a simulator should be used. Furthermore, recognizing the constrained fiscal environment and pressure on defense accounts, the strategy lists 12 investment priorities, including investing in simulators and simulations that have the greatest potential to generate cost savings, and it assigns responsibility for developing a methodology for tracking return on simulator investments.

The Army and Marine Corps Cite Benefits of Simulation-Based Training, but Lack Information to Evaluate Impacts on Performance and Cost

Army and Marine Corps training documents and officials from both services have noted benefits from the use of simulation-based training — both in terms of training effectiveness and in terms of cost savings or cost avoidance. According to various documents from both services, training in a simulation-based environment complements necessary live training and allows the fundamentals to be practiced in a more cost-effective manner. Further, simulation-based training allows servicemembers to quickly replicate a wide variety of conditions, receive instant feedback on performance, and conduct multiple iterations to master a task. In addition, training with simulation-based devices can improve training efficiency by: controlling costs by expanding the number of training sites; reducing training time; improving safety and reducing equipment wear and tear; reducing or offsetting training ammunition requirements and/or operating tempo costs; and reducing the need for additional training land. Finally, training officials from both the Army and the Marine Corps noted the improved performance of servicemembers in a live training environment as a result of the increased use of simulation-based training. However, the services lack information to assist them in better determining how the use of simulation-based training devices contribute to improved performance and evaluating the costs of simulation-based training.

While the Army and Marine Corps currently collect data on the usage of simulators and the Army is conducting a study to gain insight into how simulation-based training contributes to training, neither service has established metrics or indicators to assist them in more broadly measuring the impact of simulation-based training on improving the performance or proficiency of servicemembers. As the services look at ways to optimize the use of training resources, such information could be useful to guide decisions on the optimal mix of live and simulation-based training during the training development process, and as commanders exercise discretion in how best to integrate the use of simulators during home station training. Specifically, we found that the services collect utilization data to manage the scheduling and distribution of simulation- based training devices across installations to support home station training. For example, on a monthly basis, both the Army and Marine Corps collect data on the number of hours simulators are used and throughput (i.e., the number of soldiers or marines who use a simulator), and in some cases, the number of virtual rounds fired or miles driven. Further, in the case of the Army, the services’ training materiel developer is currently conducting a study to determine the benefits of simulation- based training. Specifically, the Program Executive Office for Simulations, Training, and Instrumentation, is compiling case studies to demonstrate the benefits and impacts of training certain tasks using simulations. The intent of this study is to evaluate how training with simulation-based devices compares to training without these devices. The Army’s study, which it expects to release later in 2013, as well as the utilization data both services are collecting, could be useful in establishing metrics to help optimize the services’ use of their training resources. Neither service has taken steps to identify performance metrics and the type of performance data that would be needed to evaluate how the use of simulation-based training devices contributes to training effectiveness. Officials told us they recognize the value of performance metrics, but given the pace of operations in Iraq and Afghanistan over the past several years, priority was focused on conducting training and preparing forces to deploy. Officials further noted that some training tasks are subjective, making it difficult to develop specific, quantifiable metrics, and that different devices may require different sets of metrics to reflect how their use contributes to improved performance. Currently, in the absence of performance data, the services obtain information on the contribution of simulation-based training from subject matter experts, who are responsible for developing training programs of instruction and overarching strategies, as well as information based on feedback and after action reports from deployments and training exercises.

As previously noted, the Navy has identified guiding principles for simulator use, including a principle that notes the need to quantifiably demonstrate how simulator use contributes to achieving training objectives. Further, leading management practices recognize, when designed effectively, performance measures help decision makers (1) determine the contributions that training makes to improve results, (2) identify gaps in performance, and (3) determine where to focus resources to improve results. In particular, incorporating valid measures of effectiveness, i.e., outcome measures, into training programs would enable an organization to better ensure that desired changes will occur in trainees’ skills, knowledge, and abilities. We recognize and have previously reported that it is difficult to establish performance measures for outcomes that are not readily observable.to determine how the use of simulation-based training devices contribute to improved performance, decision makers in the Army and Marine Corps lack information to make informed decisions about the optimal mix of training.

GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: Nov. 1, 1999).

Force’s use of live and virtual training, for instance, we reported that the Air Force had estimated it could realize savings in its training program by reducing live flying hours and taking other steps, such as increasing the use of virtual training. However, the Air Force did not have a complete picture of costs related to virtual training. For example, it had excluded certain costs from its estimates, such as expenses for aircrew travel to simulator locations, additional contractor personnel to schedule and operate simulators, and the purchase of additional simulators to meet increased demand. Therefore, we recommended that the Air Force develop a methodology to determine the universe of costs and a means to collect and track data in order to enhance its ability to make future investment decisions about the mix of live and virtual training. DOD concurred with our recommendation and noted that the Air Force is developing a standard methodology of accounting and tracking cost categories associated with live and simulation-based training.

In general, the Army and Marine Corps collect and assess some costs associated with the use of simulation-based training devices as part of their acquisition and budgeting processes. For example, both services conduct cost benefit analyses—including a review of estimated simulator and simulation life-cycle costs, such as development costs and costs to dispose of a device at the end of its life—as they make acquisition decisions such as whether to develop new simulation-based training devices. In addition, after the simulator or simulation is acquired and fielded, the services consider the life-cycle cost to operate and maintain them as they identify the funding needs to be considered in the budget development process. However, they do not reevaluate cost information during their training development process as they determine which training should be conducted live and which can be conducted using a combination of live and simulation-based training. Further, at this point in time, neither service has a methodology for identifying the universe of costs associated with using simulation-based devices or a means to collect and track these costs.

According to Army and Marine Corps officials, additional cost information would be useful in making decisions on the mix of training and related investment decisions. During the course of our work, some officials cited examples of specific costs that could be considered and variables that might need to be taken into account in developing an approach to comparing simulation-based and live training. For example, some officials noted that ammunition costs could be considered significant to both types of training, but that there are some variables to be considered in comparing these costs. For instance, they noted that the number of virtual rounds fired in a simulator could be compared to the cost of live ammunition, but that there are limits to this type of comparison because units can fire more rounds in a simulator than they would be allocated during live training. As a result, a one-to-one comparison does not present a totally accurate picture of the potential costs that are saved or avoided when conducting simulation-based training rather than live training. Additionally, they noted that ammunition has a shelf life. If it is not used within its available shelf life the services can incur costs to demilitarize the ammunition. Therefore, firing live ammunition that is approaching the end of its shelf life may actually result in cost avoidance rather than an additional cost for the services. In addition, they noted that comparing the costs of fuel and spare parts of an aircraft used in live training to the cost of technicians and spare parts needed to maintain simulators would not provide an accurate comparison if the costs of facilities, utilities, and training personnel to support the simulators were not included as well. These costs and variables could serve as the foundation for developing a cost methodology. Without a means to assess the impact of using simulators on performance and to compare the costs associated with live training and the use of simulation-based training devices, decision makers in the Army and Marine Corps lack information to make fully informed decisions in the future regarding the optimal mix of training and related investment decisions.

Conclusions

As the Army and Marine Corps take steps to further integrate the use of simulation-based training with live training, and collaborate on development efforts in a fiscally constrained environment, both services are facing important decisions regarding how to adapt current approaches to meet training requirements and prioritize related investments, including those related to the acquisition of simulation-based training capabilities. While both services have noted benefits from the use of simulation-based training —in terms of training effectiveness and in cost savings or cost avoidance, it is important that they have valid performance and cost data to assist them in evaluating these benefits. We recognize that both services currently collect various types of information on the use of simulation devices and consider costs to a certain extent in their acquisition and budgeting processes. However, taking additional steps to expand these efforts by establishing performance-oriented metrics and a methodology to identify the costs associated with simulation-based training would provide them greater insights into how the use of simulation-based training contributes to improved performance or proficiency of servicemembers, and a point of comparison for assessing the cost implications of using simulation-based or live training. Moreover, until the Army and Marine Corps take actions to increase their visibility over the impact of simulation-based training on performance and costs, they will continue to lack key information that could assist them in determining how to optimize the mix of live and simulation-based training in the future and target simulation-based training investments on the devices that have the greatest potential to improve mission performance.

Recommendations for Executive Action

To improve decision makers’ abilities to make fully informed decisions concerning whether training requirements can be met with live and simulation-based training and determine optimal mixes of live and simulation-based training, we recommend that the Secretary of Defense direct the Secretary of the Army and the Commandant of the Marine Corps to take the following two actions:

Develop outcome-oriented performance metrics that can be used to assess the impact of simulation-based training on improving the performance or proficiency of servicemembers and units.

Develop a methodology—to include identifying the costs that should be included and how these costs should be captured—for comparing the costs associated with the use of live and simulation-based training.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD partially concurred with our recommendations. In response to our recommendation that the Secretary of Defense direct the Secretary of the Army and the Commandant of the Marine Corps to develop outcome-oriented performance metrics that can be used to assess the impact of simulation- based training on improving the performance or proficiency of servicemembers and units, DOD agreed that an enhancement in outcome-oriented performance metrics would be helpful in the decision- making process. DOD noted that given the magnitude and scope of training tasks, varying competencies of the training audience, and ever changing technology, the problem set contains many independent variables. DOD said that it will study the problem set, and as appropriate, develop a construct and implementation plan to include performance metrics to assess the impact of simulation-based training on improving the performance or proficiency of servicemembers and units. We recognize that DOD must consider many independent variables and note in our report that some training tasks are subjective, making it difficult to develop specific, quantifiable metrics. However, facing these same types of challenges, we describe in our report that the Navy has a simulator strategy that notes the need to quantifiably demonstrate how simulator use contributes to achieving objectives. Further, as noted in our report, Army and Marine Corps training tasks already have associated conditions and standards. These standards could be used as the basis for developing performance metrics to evaluate differences, if any, between live and simulation-based training effectiveness. As both services have noted anecdotal benefits from the use of simulation-based training, we continue to believe that establishing performance-oriented metrics would provide the Army and Marine Corps with greater insights into how the use of simulation-based training contributes to improved performance or proficiency.

DOD partially concurred with our recommendation that the Secretary of Defense direct the Secretary of the Army and the Commandant of the Marine Corps to develop a methodology—to include identifying the costs that should be included and how these costs should be captured—for comparing the costs associated with the use of live and simulation-based training. In its comments, DOD noted that the Army and Marine Corps capture all relevant costs needed for decision-making during the Planning, Programming, Budgeting, and Execution (PPBE) process in procuring simulators/simulation devices. DOD further stated that the Marine Corps training is based on the Systems Approach to Training, which includes policy on developing outcome-oriented performance metrics that are employed in school house lesson plans and home station training. DOD noted that the combination of PPBE and the Systems Approach to Training ensures that costs are considered in determining the mix of live and simulation-based training. DOD further stated that the Army and Marine Corps concur that a more comprehensive cost analysis would assist decision making in determining the optimal mix of live and simulation-based training. DOD did not cite any specific steps that the services plan to take. In our report, we specifically recognize that the Army and Marine Corps assess costs associated with simulation based devices, such as life cycle costs, as they make acquisition decisions and during their budget development process when they determine funding needs to operate acquired devices. However, the services do not reevaluate cost information during their training development process which is the point at which they are determining the mix of live and simulated based training. Furthermore, beyond those costs currently assessed in the budget process, we found examples of additional costs that could be considered if the services were to perform a cost comparison analysis between live and simulated based training, such as facilities, fuel and ammunition. Therefore, to enhance their ability to optimize the mix of training and better understand related cost implications, we continue to believe the services need to expand their current efforts and take specific steps to develop a methodology for comparing costs associated with the use of live and simulated based training. DOD’s comments are included in their entirety in appendix II. DOD also provided technical and clarifying comments, which we have incorporated where appropriate.

We are sending copies of this report to appropriate congressional committees, the Secretary of Defense, the Secretary of the Army, and the Commandant of the Marine Corps. In addition, this report will be available at no charge on our website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To address our objectives, we met with officials from the Office of the Secretary of Defense, the Department of the Army Headquarters, Marine Corps Headquarters, and several Army and Marine Corps commands and organizations, and visited simulation-based training facilities. Our review focused on the mix of live and simulation-based training for institutional and home station training because this training represents a significant portion of both services’ training. Excluded from this review were live and simulation-based training at the services’ combat training centers and during deployment. We selected Army and Marine Corps occupations that use the largest number of simulation-based training devices and represent a broad cross-section of how these devices are used by the services, respectively. For both the Army and Marine Corps, we selected the aviation, armor, and artillery occupations. In addition, for the Army we included infantry, and for the Marine Corps we included amphibious assault vehicles and motor transport occupations. We held discussions with training officials representing each of these occupations. In addition, we visited the Army Aviation Center of Excellence in Fort Rucker, Alabama; the Army Maneuver Center of Excellence at Fort Benning, GA, where both services train their armor personnel; and Fort Sill, Oklahoma, where both services train their artillery personnel.

To determine how the Army’s and Marine Corps’ use of simulation-based training devices has changed since the services first began using simulators, we reviewed and analyzed service briefings and documentation that provided information on the historical use of simulation-based training devices, and the timelines within which simulators for various occupations became available. In addition, we interviewed officials from the Department of the Army-Management Office for Training Simulations; Marine Corps Plans, Policies and Operations; and the services’ training commands —the Army’s Training and Doctrine Command and the Marine Corps’ Training and Education Command—to discuss how simulators and simulations were and are currently being used. We also interviewed officials representing the selected service occupations and from the Army Reserve Command, the Army National Guard, and Marine Corps Forces Reserve, to discuss the mix of live and simulation-based training, how the use of simulators and simulations has evolved, and the benefits, limitations, and challenges of simulation-based training. Additionally, we reviewed our prior reports to gain additional perspective on how simulators and simulations were used in the past. To assess the Army’s and Marine Corps’ efforts to better integrate live and simulation-based training, we evaluated documentation on simulation- based training technologies and capabilities. In addition, we met with officials from the Army’s and Marine Corps’ training commands and materiel development organizations—the Program Executive Office for Simulations, Training, and Instrumentation and the Program Manager for Training Systems, respectively—to discuss how they coordinate the development of simulation-based training capabilities and technologies and ongoing initiatives to further integrate simulation-based training. We also obtained and reviewed guidance and documentation from both services related to the development of certain technical capabilities to connect incompatible simulation-based training devices—the Army’s Live, Virtual, and Constructive—Integrated Architecture (LVC-IA), and the Marine Corps’ Live, Virtual and Constructive Training Environment. We discussed these technical capabilities with officials from the Army headquarters, and the services’ training commands and materiel development organizations. In addition, we reviewed Army documents on the LVC-IA and the summary of the findings and lessons learned from the initial fielding of the integrated architecture at Fort Hood, Texas. We also interviewed officials from Fort Hood, including company, battalion, and brigade-level officials from the 2nd Brigade Combat Team, 1st Cavalry Division, to discuss their perspectives on the benefits and challenges related to the initial LVC-IA exercise. Additionally, we reviewed Marine Corps’ documents, such as its Live, Virtual and Constructive Training Environment Initial Capabilities Document; and the Training and Education Modeling and Simulation Master Plan 2010. We also interviewed Marine Corps training officials to obtain information on the purpose and current state of its Live, Virtual and Constructive Training Environment.

To ascertain the factors the Army and Marine Corps consider in determining whether to use live or simulation-based training, including the extent to which they consider performance and cost information, we assessed Army and Marine Corps documentation, such as the 2013 Army Posture Statement; U.S Army Training Concept 2012-2020; Army Regulation 350-38, Policies and Management for Training Aids, Devices, Simulators, and Simulations; the 2012 Army Training Strategy; the Posture of the United States Marine Corps, 2013 Report to Congress; and Marine Corps Concepts and Programs 2013. We reviewed Department of Defense (DOD) and Navy Training guidance. We also reviewed our previous reports on Air Force and Navy virtual training. To determine the services’ mixes of live and simulation-based training for institutional and home station training, we reviewed Army and Marine Corps policies and guidance related to developing and conducting training, including the Training and Doctrine Command Regulation 350- 70: Army Learning Policy and Systems; U.S. Army Training and Doctrine Command Pamphlet 350-70-1, Training Development in Support of the Operational Domain; and NAVMC 1553.1: the Marine Corps’ Systems Approach to Training User’s Guide. We interviewed officials from the services’ training commands, the Army Reserve Command, the Army National Guard, Marine Corps Forces Reserve, and subject matter experts for the selected occupations to discuss the mix of training at institutions and home station for active and reserve component personnel, how these mixes are developed, and the factors that are considered. We reviewed programs of instruction to obtain examples of the mixes of live and simulation-based training prescribed during institutional training in the selected occupations. To determine the services’ mix of training at home station we reviewed examples of the Army Combined Arms Training Strategies, Army proponent’s Home Station Gated Training Strategies, and the Marine Corps’ Training and Readiness Manuals. We also interviewed officials from the Army Reserve and Army National Guard, as well as unit commanders and officials from the installations that we visited, i.e., Army’s 3rd Brigade, 3rd Infantry Division at Fort Benning, Georgia; 1st Air Cavalry Brigade, 1st Calvary Division, at Fort Hood, Texas; 31st Air Defense Artillery Brigade, 75th Fires Brigade, and 214th Fires Brigade at Fort Sill, Oklahoma; and former Marine Corps battalion and company commanders and officials from the Marine Corps Forces Reserve to discuss how the Army’s and Marine Corps’ overarching training strategies assist commanders in developing a mix of live and simulation-based training for home station training. In addition, we interviewed Army and Marine Corps training officials to identify and determine the types of information collected on the use of simulators and simulations; how this information was being used; and how the services determine the relationship between simulator usage and performance. We also obtained examples of the types of information being collected by both services, and participated in an online demonstration of the Army’s central repository for this information. In addition, we reviewed the Navy’s Overarching Fleet Simulator Strategy and management practices on performance measures. To determine how costs are considered and compared when developing live and simulation- based training, we met with officials from Department of the Army and both services’ training commands. We reviewed the Army’s 3rd edition of its Cost Benefit Analysis Guide to identify potential DOD criteria as to what factors should be included when conducting a cost-benefit analysis. We also reviewed Army and Marine Corps acquisition guidance, which provides instruction on conducting cost-benefit analyses to acquire simulators. In addition, we reviewed federal internal control standards, our past reviews on the Air Force’s use of live and virtual training, and federal cost-estimating and budgeting guidance. Further, we interviewed training officials and current and former unit commanders from both services to further clarify how costs are considered when developing the different mixes of live and simulation-based training.

In conducting this work, we contacted officials from the organizations outlined in table 2.

We conducted this performance audit from June 2012 to August 2013, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Sharon Pickup, (202) 512-9619 or pickups@gao.gov.

Staff Acknowledgments

In addition to the contact named above, key contributors to this report were Michael Ferren, Assistant Director; Richard Burkard; Kenya Jones; Jeff Rankin; Michael Silver; Susan Tindall; Erik Wilkins-McKee; and Richard Winsor.