Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 1 of 11

Assessing the Effects of Virtual Emergency Training on 

Mine Rescue Team Efficacy 

Hoebbel C, Bauerle T, Macdonald B, Mallett L

The National Institute for Occupational Safety and Health

Pittsburgh, PA

choebbel@cdc.gov, tbauerle@cdc.gov, bmacdonald@cdc.gov, lmallett@cdc.gov

ABSTRACT

To reach trapped miners, underground mine rescue teams might be required to perform a variety of non-routine tasks 

(e.g., fight fire, pump water, support unsafe roof) as they encounter hazardous and rapidly changing conditions. Due 

to risks associated with such conditions, underground mine rescue team training has traditionally been performed 

using live exercises in the form of competitive drills in above-ground facilities or open fields. Oftentimes these 

contests utilize printed paper placards to represent environmental conditions and have strict rules which were 

developed for comparative assessment purposes. Although widely used, these contests have undetermined ‘real world’ 

application, are low fidelity, and have limited documented evidence for effectiveness. Both miners and subject matter 

experts have emphasized the need for more realistic and engaging training environments to enhance the learning 

experience of all miners and emergency responders. To this end, the National Institute for Occupational Safety and 

Health (NIOSH) has developed a fully immersive dynamic virtual training environment for mine emergency

responders. During training scenarios, rescue teams approach the problem and perform as they would in “real life” 

utilizing virtual representative equipment, with success dependent on effective communication and group decisionmaking. This study represents the first documented empirical attempt at evaluating the effectiveness of virtual reality 

training for mine rescue teams in the United States, and this paper discusses the utility of such environments for not 

only delivering realistic and engaging training, but for conducting behavioral research activities. Associations among 

psychosocial factors such as training climate, team familiarity, and team efficacy are examined and, in general, the 

study results support findings and recommendations found in the emergency teamwork literature. The results of this 

effort will add to the research base on mine emergency response training and assessment as well as provide insights 

into emergency response team behavior.

ABOUT THE AUTHORS

Cassandra Hoebbel is a Behavioral Scientist in the Human Factors Branch for the National Institute for Occupational 

Safety and Health (NIOSH) Office of Mine Safety and Health Research (OMSHR). Her research focuses on mine 

emergency response systems. Cassandra received her B.A. in Psychology from the State University College at Buffalo, 

and her Ed.M. and her Ph.D. from The Graduate School of Education at the State University of New York at Buffalo. 

Tim Bauerle is a Behavioral Scientist in the Human Factors Branch at NIOSH OMSHR. He has a master's degree in 

Industrial and Organizational Psychology from the University of Connecticut, with concentrations in Occupational 

Health Psychology and Quantitative Research Methodology. His research primarily focuses on safety climate and risk 

perception in hazardous occupations and he expects to complete his doctoral degree in 2016.

Brendan Macdonald is an Associate Service Fellow at NIOSH OMSHR. His research focuses on the development 

and application of virtual reality technologies within the context of mine safety and health. Brendan’s current projects 

focus on interactive visualizations and the use of complex interaction methods for multiple guests within a simulation. 

Brendan received his B.S. in Mechanical Engineering from Vanderbilt University and his master’s degree in 

Entertainment Technology from Carnegie Mellon University.

Launa Mallett is a Lead Behavioral Scientist at NIOSH OMSHR. She holds an M.A. in Anthropology and a Ph.D. 

in Sociology from the University of Kentucky. Dr. Mallett’s research interests include mine emergency response and 

workplace safety and health training techniques and technologies.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 2 of 11

Assessing the Effects of Virtual Emergency Training on 

Mine Rescue Team Efficacy 

Hoebbel C, Bauerle T, Macdonald B, Mallett L

The National Institute for Occupational Safety and Health

Pittsburgh, PA

choebbel@cdc.gov, tbauerle@cdc.gov, bmacdonald@cdc.gov, lmallett@cdc.gov

INTRODUCTION 

Throughout the course of the 20th century, the number of catastrophic mining events steadily declined in both number 

and scale. Because there was only one underground disaster in the 1990s (South Mountain Coal Company No. 3 Mine 

in 1992, 8 fatalities), it was possible to hold the view that emergency response no longer needed to be a top research 

priority. New miner and annual refresher training emphasized required routine knowledge, skills, abilities, and other 

attributes (KSAOs) over the non-routine KSAOs required to respond to seemingly unlikely large-scale emergencies. 

Then came Brookwood in 2001 (13 fatalities); followed by Quecreek (2002, 0 fatalities); Alma No. 1(2006, 2 

fatalities), Sago (2006, 12 fatalities), Darby (2006, 5 fatalities), Crandall Canyon (2007, 6 fatalities), and Upper Big 

Branch (2010, 29 fatalities). These tragedies brought the critical need for emergency preparations back to the forefront

and the mining industry and state and federal government agencies reacted. These incidents highlighted the need to 

balance investments that would reduce low-probability/high-severity events with those that focus on more frequent, 

but less severe injuries and illnesses. The Mine Improvement and New Emergency Response Act (MINER Act) of 

2006 was passed in an attempt to help prevent future disasters and to specifically address and increase training 

requirements. This presented new opportunities for the research and development of training in non-routine KSAOs 

required to respond to large-scale emergencies.

The current minimum federal regulations (30 CFR Parts 49 and 75) for mine rescue team training require: 

• 96 hours annually 

• Participation in 2 local mine rescue contests 

o Company team – annually at each mine covered by the rescue team

o Composite teams – semi-annually at each mine covered by the rescue team

o Contract team – quarterly at large and semi-annually at each small mine covered by the rescue team

o State sponsored team – annually at each mine covered by the rescue team 

• Annual participation in smoke training 

• 2 hours wearing breathing apparatus under oxygen every 2 months

While federal mandates for frequency of training are important for mine rescue team member skill development, the 

benefits are difficult to assess, and modern approaches to training in a virtual environment can offer promising 

potential benefits. The goal of this research effort was to assess the benefit of immersive mine rescue team training on 

emergency-based teamwork and its potential implications. In addition to the more concrete task-oriented knowledge 

and skills required for effective mine emergency response, this paper discusses other salient team-based constructs 

such as training climate, team familiarity, and team efficacy. 

BACKGROUND 

Mine Rescue Teams 

The Federal Mine Safety and Health Act of 1977 (Public Law 91-173, as amended by Public Law 95-164) required 

mine rescue teams for all underground mines in the United States. While all miners must prepare for emergencies, 

mine rescue teams form the core of the on-the-ground response when a large-scale event occurs. Rescue team members 

must be prepared to enter the complex and dynamic environment that exists in the aftermath of a catastrophic event. 

According to the Mine Safety and Health Administration’s (MSHA) “Mine Rescue Teams Nationwide” webpage

(http://www.msha.gov/MineRescue/MAP/ASP/minerescuehome.asp), there were 436 underground mine rescue 

teams (215 coal and 221 metal/nonmetal) in the United States as of March 2014, with 1875 individuals serving on 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 3 of 11

underground coal teams and 2004 on metal/nonmetal teams, totaling 3879 underground mine rescue team members 

from 33 states. 

Large mine companies sometimes form teams made up of employees, while smaller mines might contract with 

professional teams, rely on state teams, or come together with neighboring small mines to form composite teams. 

MSHA and a number universities have also assembled their own mine rescue teams. While MSHA sets the minimum 

standard for training, many mining companies exceed this with internal policies and standards. Due to these and many 

other factors, mine rescue teams possess highly variable mine emergency response experience and skills.

Mine Rescue Team Training

Mine rescue training began in the United States in 1910 and quickly evolved into local and regional competitions that 

are still held today. Due to the risk associated with training in dangerous environments, these contests have historically 

been performed in above-ground facilities or open fields like the one depicted in Image 1. While these competitions 

have been widely accepted by the underground mining industry and participation is mandated by law, there is little 

evidence to support (or controvert) their effectiveness, and the industry is lacking evidence-based guidance on how 

best to prepare mine rescue teams to respond to emergency situations. 

Image 1. Traditional Mine Rescue Contest Field

When federal law mandated training for all mine rescue teams in the U.S. in 1977, the U.S. Bureau of Mines, then 

later the National Institute for Occupational Safety and Health’s (NIOSH) Office of Mine Safety and Health Research, 

began to support the design and development of more realistic and engaging training materials and mine emergency 

simulation exercises. Many of these simulations were based on actual emergency events reported in formal 

investigations by MSHA, and on firsthand accounts of miners who were directly involved in the events. Built on 

unfolding narratives, training activities were designed to teach the critical problem-solving, decision-making, and 

other non-routine KSAOs required for mine rescue teams to effectively respond to emergencies. Most of the early 

exercises involved small group discussions within classroom settings and/or the use of tabletop simulations that

allowed trainees to actively engage in problem solving. More recently, more advanced technologies have been 

introduced to add more realism to the training environment. 

Technology-based Training

Extensive research on occupational safety training methods suggests that two of the most important attributes of 

effective training are high engagement and realism (e.g., Burke et al., 2006, Robson et al., 2012). The use of 

technology-based simulations to deliver realistic and engaging training is not new. In fact, the patent for the Link 

Trainer for pilot training was applied for in 1929 (Amico, 2009), and over a decade ago Don Johnson, a member of 

the Readiness and Training Unit of the Department of Defense (DoD) at the Pentagon, was quoted as saying, “In DoD, 

we probably have done more studies of learning technology than you will find anywhere. We’ve proven to ourselves 

that technology works. We’ve proven it academically, but more importantly, we’ve proven it operationally” (Prensky, 

2001). 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 4 of 11

The mining industry has long accepted equipment simulators that incorporate virtual reality components, and virtual 

environments are currently being used for underground mine health and safety training in several countries. Initial 

indications are that this form of training adds significant realism and is working quite well for health and safety training 

in the mining industry (Galvin, 2008). In lieu of full-scale drills, Australia has been staging parts of mine rescue 

competitions in immersive 360° virtual environments since 2010 (http://www.minesrescueservices.com). Such 

immersive environments are supported by advanced simulations that are capable of reproducing the underground 

layout and physical conditions of mines and allowing for the observation and evaluation of team behavior. NIOSH

established its own Virtual Immersion and Simulation Lab (VISLab) in 2012 and it is the first such facility built in the 

United States (see Methodology section for a brief description). 

NIOSH has experienced early success using virtual technologies for simulation, and preliminary data suggest these 

simulated environments offer both the realism and engagement levels that are critical for effective training

(http://www.cdc.gov/niosh/mining/topics/RescueTechnologiesandTraining.html). Results from the present study 

bolster these findings and support the argument that these environments can also be used to observe and assess team 

dynamics relevant to successful emergency response team function. 

Team Training

Schaafstal et al. (2001) summarized the role and captured the state of emergency groups and teams best: “[Emergency 

teams] are often faced with ill-structured problems, uncertain and dynamic environments, shifting, ill-defined and 

sometimes competing goals, multiple event-feedback loops, time constraints, high stakes, multiple players, and 

organizational norms and goals that must be balanced against the team members’ personal choices” (p. 617). For this 

reason, a great deal of the literature on emergency groups and teams focuses on team process elements (i.e. the process 

that teams employ to achieve performance) that can predict performance in non-routine real-world or simulated 

emergency events. Schaafstal et al. (2001) explicitly emphasize teamwork over task work, arguing that tasks in 

emergency scenarios are highly fluid and context dependent. To this end, the authors cite US Navy research on the 

Tactical Decision Making Under Stress (TADMUS) training framework (e.g., Cannon-Bowers & Salas, 1998), which 

focuses on teamwork dimensions that can distinguish expert from novice teams, such as information exchange, 

communication, and supporting behavior. Similarly, Serfaty et al. (1993) call attention to the notion that a team’s 

ability to work together and adapt to changing environmental demands is paramount for team performance, especially 

among emergency teams. Other evidence suggests that teams with fluid tasks achieve better performance outcomes 

when teams have greater group participation (Linehan et al., 2009); shared mental models, collective efficacy, implicit 

coordination, and adaptive strategy (Waller, Lei & Pratten, 2014); greater familiarity (Huckman & Staats, 2011) and 

more consistent backup behavior (Smith-Jentsch et al., 2009).

This body of literature seems to suggest that optimal training for emergency teams should focus on teamwork elements 

critical for successful performance, ideally through practice under a variety of simulated emergency conditions. As 

Ford and Schmidt (2000) put it, “With emergency training...the assumption that the KSAs being trained will be used 

or applied immediately on the job does not hold. A key issue, then, is how to design training programs and how to 

develop on the job activitiesthat help individuals not only maintain current knowledge and skill levels but also enhance 

their knowledge and skills given limited or nonexistent opportunity to perform trained tasks directly on the job” (p. 

200). Given this limitation, virtual training environments have the capacity to provide knowledge and skill acquisition 

opportunities and can lend themselves to the evaluation of critical non-routine KSAOs and observations of team-based 

dynamics that are believed to influence “real world” team performance. 

METHODOLOGY

Target Population

The population for the study included underground mine rescue teams who were recruited by NIOSH OMSHR through 

mining organization contacts, team captains, trainers, and other stakeholder relationships. In 2014, mine rescue teams 

from Southwestern Pennsylvania and West Virginia visited the NIOSH Bruceton Research Site near Pittsburgh, PA, 

to voluntarily participate in this study. In total, 70 participants representing 10 mine rescue teams with variable 

composition and a wide range of experience participated.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 5 of 11

Simulation Environment

NIOSH OMSHR’s Virtual Immersion and Simulation 

Laboratory (VISLab) is a multipurpose training research and 

development lab for virtual reality applications and uses state-ofthe-art technology for training and assessing miners. Through 

observation and the use of interaction devices, researchers have the 

ability to capture data within training simulations that would be 

impossible to replicate in real life. The facility includes a state-ofthe-art 360° cylindrical projection system that uses stereoscopic 3D 

technology designed to provide an intuitive training experience for 

participants (Image 2-A). Surrounding the screen is a 7.1 channel 

surround sound system to aid in providing directional auditory cues 

as dictated by the rescue problem. An adjacent 50° curved screen 

display acts as a classroom environment for instructor-driven sessions and serves as a post-simulation debriefing 

environment for after-action reviews (Image 2-B). Room C in Image 2 serves as a fresh air base for the team briefing 

officer.

Image 2. Overhead representation of

OMSHR’s VISLab

Image 3. Mine Rescue Trainee under Apparatus Image 4. After-Action Review

Instrument Development

The primary goal of this effort was to develop and assess an immersive training environment that requires trainees to 

utilize the technical non-routine KSAOs required for successful emergency response, while also allowing teams to 

practice and develop ‘soft’ teamwork skills and enabling researchers to examine a variety of constructs that would be 

difficult to measure in other settings. In general, a training assessment strategy as outlined in Holton (1996; 2005), in 

which the effectiveness and outcomes of the training are evaluated alongside influencing elements – such as 

motivation, environmental factors, and abilities – was employed. While the full details of the training evaluation model

are not included here, the analysis will focus on the central teamwork elements (collective efficacy, familiarity, and 

training climate) of the assessment. The development of the instruments used to assess the effectiveness of the training 

from a teamwork perspective are described below.

In an attempt to create a theoretical model that might summarize the structural and process needs of teams, Salas, 

Sims, and Burke (2005) proposed a “Big Five” framework (hereafter referred to as “Big 5”) for teamwork which aims

to condense teamwork activities into five core concepts (team leadership, mutual performance monitoring, backup 

behavior, adaptability, team orientation) as well as three supporting coordinating mechanisms (shared mental models, 

closed-loop communication, mutual trust), with the general understanding that – regardless of task type – teams that

are successful across these process dimensions are more effective and productive when compared to their counterparts. 

Salas et al. (2005) argue that theirs is the first model, derived from an extensive review of the previous 20 years of 

research on teams that attempts to comprehensively capture teamwork. Although the framework certainly has 

appealing face value, the empirical benefits of assessing the Big 5 are unclear given the unique and diverse nature of 

emergency teams, and the reality that emergency teamwork behavior can be difficult to assess. 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 6 of 11

One solution comes from Collins and Parker (2010), whose research distinguishes team outcome efficacy and team 

process efficacy, in which the former is the team’s belief that it can achieve some level of desired outcome; whereas 

the latter is the team’s belief that it can successfully engage in the processes that are necessary to achieve this outcome. 

Although self-report methodologies come with limitations, there is the potential for assessing an emergency team’s 

teamwork KSAOs using a process efficacy approach. To our knowledge, no previous study has attempted to 

quantitatively assess teamwork in emergency teams using Salas et al.’s (2005) Big 5 framework, and survey items 

were developed with this in mind. Survey methodology espoused by Hinkin (1995; 1998) was used as a general 

framework to develop the Team Process Efficacy scales. Items were generated by reviewing the Big 5 of teamwork 

as described by Salas et al. (2005), as well as a review of the teamwork taxonomy literature (e.g., Marks, Mathieu, & 

Zaccaro, 2001). Items were reviewed by internal scale development and mining engineer subject matter experts to 

determine adequate fit and face validity. A list of ten items was logically derived and covered all five teamwork 

dimensions and three coordinating mechanisms (see Table 1). Efficacy response scales as reported by Collins and 

Parker (2010) were added to the survey, and study participants were asked to describe their sense of efficacy in the 

team by assigning a percentage between 0 and 100 percent to indicate the degree of confidence they had in their team’s

process and ability to achieve desired outcomes. All efficacy measures were collected before and after the training in 

order to detect any potential effect of training on team efficacy.

Two additional constructs (Familiarity and Training Climate) relevant to this analysis and believed to moderate gains 

in perceptions of team efficacy as a result of training were added to the assessment. It was hypothesized that mine 

rescue teams who reported more familiarity with one another would have a greater sense of team efficacy. Team 

cohesion literature suggests that teams with members who are highly familiar with one another appear to perform 

better in ambiguous circumstances (Gruenfeld, Mannix, Williams, & Neale, 1996). To measure familiarity, three items 

were created which asked trainees how often their team members see each other at work or outside of work and how 

often training across team roles occurs. Familiarity items were administered on a five-point frequency scale, from 

“Not at all” to “Frequently or always.” In addition, given the importance of both organizational climate and individual 

attitudinal measures on training effectiveness (e.g., Salas & Canon-Bowers, 2001), participants also responded to 

items from the Mine Emergency Preparedness Climate scale (Bauerle & Mallett, 2013). For this analysis, the focus is 

on the Training Climate subscale which aims to assess the degree to which organizational and environmental supports

exist and represent the enacted values of miners’ respective mine sites. 

Research Protocol/Design 

Upon arrival, all trainees were asked for their voluntary consent to participate in a mine rescue training scenario in the 

VISLab’s 360° theater and to participate in IRB-approved assessments of their knowledge and skills, levels of 

confidence, their perceptions regarding the realism of the scenario and the environment, and their level of engagement 

with the problem. 

After a short orientation program and ample opportunity to become familiarized with the virtual environment and 

interaction devices, teams were formally briefed by an experienced mine rescue trainer. The briefing included 

information about the virtual mine including details about the mine environment (e.g., mine layout, ventilation, 

escapeway routes, locations of refuge alternatives and self-contained self-rescuers (SCSRs), etc.). Teams participated 

in one of two training modules, each designed within common underground coal mining environments (i.e. room and 

pillar and longwall methods). Following the briefing, team roles were assigned as they are for actual exploration to

include a captain, map man, gas man, stretcher bearer, and tail captain. A sixth team member, known as a briefing 

officer, remains in a fresh air base and interacts through two-way radio with the team captain to track the team’s 

location and advise team movement. Once inside the room, participants navigated through the mine while accessing 

and utilizing virtual items such as fire extinguishers, SCSRs, gas meters, tether lines, and other equipment while 

remaining in radio contact with the briefing officer. The simulation typically lasted between 60 and 90 minutes, 

depending on the module and the strategies the participants chose to employ while working through the problem. 

After completion of the virtual mine rescue scenario, participants underwent a 30-minute after-action debriefing 

session during which the mine map was projected on the 50° curved screen display. This allowed the mine rescue 

trainer to review the actions and exploration paths taken by the teams, the decisions that were made, and to ask 

questions and promote discussion surrounding the team’s strategy. Following the debriefing, subjects were 

administered questionnaires designed to capture their thoughts and feelings about the face validity of the training 

exercise as well as training-specific attitudes and their reaction to the training. Participants also completed a similar 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 7 of 11

questionnaire before briefing in order to compare pre- and post-test responses. In total, participants were administered 

three brief questionnaires: one administered before briefing (“pre-training”), one immediately after the simulation 

(“post-simulation”), and one after debriefing (“post-training”). Participants were also asked to complete pre- and posttraining knowledge questionnaires. Time was also allowed for trainees to express their opinions about their experience, 

in general, and for research staff to address any questions or concerns raised by trainees. Overall, each of the mine 

rescue training sessions including the pre- and post-assessments took between 3 and 4 hours. 

RESULTS AND ANALYSIS

Sample Characteristics

Seventy participants from 10 underground mine rescue teams consisting of 6 company teams, 3 composite state teams, 

and 1 university team participated. Due to the relatively small number of teams in the study, results are presented with 

the individual as the unit of analysis. The participants were almost exclusively male (98.6%) and ranged in age from 

21–58 with a mean age of 33.22 (SD = 10.02). Participants reported an average of 9.54 (SD = 8.26) years of experience 

in mining and an average of 4.13 (SD = 5.24) years of experience on a mine rescue team. Of this group, 32.9% reported 

having experienced a virtual reality environment before participation in this study.

Participant Reactions

Based on survey results, most of the participants reacted positively to the training environment. A majority of 

participants agreed or strongly agreed that the training included relevant content to mine rescue team members (100%), 

was a good supplement to existing mine rescue training (97%), and made them more confident in correctly responding 

to a real mine emergency (94%). Furthermore, on a 7-point Likert immersion scale, most of the participants reported 

above the median rating (i.e. > 4) regarding feeling immersed in the virtual environment (68.9%), being engaged in 

the virtual reality experience (77%), and being involved in the exercise to the extent that they lost track of time 

(56.5%). Finally, 82.5% of participants reported at or above the median rating that the virtual simulated environment 

was consistent with that of an actual underground coal mine. 

Scale Development

Due to the exploratory nature of this study, principal components analysis (PCA) with varimax rotation was used to 

investigate solutions for both the process efficacy and climate measures (Fabrigar, Wegener, MacCallum & Strahan, 

1999). A final two-factor solution was discovered (Table 1) from the initial 10 team efficacy items, which accounted 

for 72% of the variance and closely mirrored Salas’ framework, with one factor representing the ‘Big 5’ processes (α= 

.88) and the other factor representing ‘Coordination Mechanisms’ (α = .85). Likewise, a parsimonious four-factor

model was discovered for Mine Emergency Preparedness Climate which accounted for 78% of the variance, and the 

Training Climate factor (α = .83) included the following items:

• Our mine rescue training has opportunities for constructive feedback from instructors.

• Our mine rescue training includes practice using the equipment we would use in a real emergency.

• Our mine rescue training is usually realistic and “hands-on”.

A repeated measures ANOVA (Table 2) determined that both the Big 5 (F(1,57) = 26.01 , p < .01, η2 = .31) and 

Coordination (F(1,57) = 10.86, p < .01, η2 = .16) aspects of perceived team efficacy improved for trainees as a result 

of training. Furthermore, with regard to the Coordination subscale, team familiarity significatly interacted with 

efficacy over time (F(1,57) = 4.02, p < .05, η2 = .07), such that members from highly familiar teams had greater gains 

in efficacy during post-test assessment (Figure 1). Additionally, for the Big 5 indicator of team efficacy, a significant 

three-way interaction was found between efficacy, team familiarity, and training climate (F(1,57) = 7.31, p < .01, η2 

= .11). As expected, members from teams with a supportive training climate and high member familiarity had the 

highest pre- and post-test efficacy ratings, while the converse was true for members from teams with low scores on 

training climate and low member familiarity. Tukey post-hoc comparisons indicated mean Big 5 efficacy scores for 

the low familiarity high training climate group did not differ significantly from the high familiarity high training 

climate or for the low familiarity low training climate groups at pre- or post-test. However, the high familiarity low 

training climate group mean score 70.29 (SD = 14.49) did differ significantly at from the high familiarity high training 

climate group mean score 85.62 (SD = 12.8) at at pre-test and this difference became statistically non-significant at 

post-test (Figure 2). 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 8 of 11

Table 1. Final Rotated Solution for Pretest Team Process Efficacy

Item Text Salas’ Framework Construct 1 2 

Readapt team strategy in response to unexpected 

changes in the environment. Adaptability Big 5 .86 .30

Make quick and effective decisions Team Effectiveness Big 5 .84 .34

Agree on solutions to difficult problems encountered. Team Orientation Big 5 .79 .35

Maintain clear and efficient communication. Closed-Loop Communication Big 5* .72 .42

Monitor each other’s work to catch mistakes and 

provide feedback.

Mutual Performance 

Monitoring; Back-Up Behavior Big 5 .66 .51

Assist with each other’s responsibilities. Mutual Performance 

Monitoring; Back-Up Behavior Big 5 .65 .13

Maintain a shared understanding of each other’s 

responsibilities. Share Mental Models Coordination .30 .84

Maintain coordination in performing the tasks at 

hand. Coordinating Mechanisms Coordination .42 .79

Consider teammate input before making an action. Mutual Trust Coordination .16 .77

Maintain a level of trust and confidence in each other. Mutual Trust Coordination .45 .74

 *This item was assigned to construct based on theoretical justification.

Table 2. One-Way Repeated Measures ANOVA with Pre- and Post-test Team Process Efficacy

Effect SS df MS F

Big 5 615.03 1 615.03 26.01**

Team Familiarity 183.25 1 185.25 7.75**

Training Climate 97.09 1 97.09 4.11*

Team Familiarity * Training Climate 172.90 1 172.90 7.31**

Error 1347.95 57 23.65

Coordination 1221.69 1 1221.69 10.86**

Team Familiarity 452.62 1 452.62 4.02*

Training Climate 81.81 1 81.81 0.73

Team Familiarity * Training Climate 129.99 1 129.99 1.16

Error 6414.61 57 112.54

Note: * p < .05 ** p < .01 

DISCUSSION

There are limited opportunities for mine rescue teams to practice coordinated use of the critical non-routine KSAOs 

required for an effective mine emergency response. Traditional full-scale drills can be cost- and resource-prohibitive 

and mine rescue team contests are strictly judged under relatively unrealistic conditions. The study results demonstrate 

that NIOSH’s VISLab offers realistic and engaging training environments for mine rescue teams to practice solving 

problems as they would in “real life.” This added element requires team members to communicate and make critical 

decisions about actions to take without regard to competitive scoring. 

As the literature shows, team training in immersive environments is important for the development of teamwork 

efficacy, and this sense of team efficacy is related to better training outcomes. Results from this study indicate that 

teams with high familiarity among members report higher team efficacy scores than low familiarity teams both preand post-training, with greater gains in coordination efficacy after training. When considering training climate with 

team familiarity, results were similar for Big 5 process efficacy.

Not surprisingly, teams reporting both low training climate and low familiarity also reported the lowest levels of 

process efficacy at pre- and post-training, and those teams reporting both high training climate and high familiarity 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 9 of 11

reported the highest process efficacy scores. Interestingly, the greatest gains in Big 5 process efficacy were observed 

in groups that reported high familiarity and low training climate. This might suggest that high familiarity teams can 

overcome other limiting factors more quickly and dramatically than their counterparts in their development of teamefficacy. All teams reported an increased sense of team efficacy post-training. 

Figure 1. Team Familiarity and Pre- and PostTraining Team Coordination Process Efficacy

Figure 2. Team Familiarity and Training Climate and

Pre- and Post-Training Team Big 5 Process Efficacy

Limitations

Due to accessibility issues with this population, it is sometimes difficult to recruit teams to visit a research laboratory.

For this reason, follow-up data collection was not part of this study design. We acknowledge the limitations this 

presents in terms of measuring any sustaining effect of the training and recognize that the extent to which performance 

in a simulated environment successfully transfers to real-world emergencies cannot be quantified. Additionally, the 

study lacks a random sample at both the mine site and the individual level. Participating mines and individual team 

members volunteered to participate and self-selection in both cases could lead to results that may not be generalizable 

across the industry. Finally, the individual data presented in this analysis was self-reported, and therefore, responses 

were subject to biases such as social desirability and acquiescence, or could be otherwise inaccurate due to concerns 

about confidentiality of responses.

Implications & Future Directions

The results of this study suggest several important considerations for training emergency response teams. First, on a 

broad level, we confirm the importance of distal indicators of training effectiveness in an emergency team training 

context. Indeed, teams who come from organizations with unsupportive training climates or teams who are less 

familiar with one another may not receive the same training benefits as their better-supported, highly familiar 

counterparts. Secondly, team familiarity emerged as a salient moderator in team efficacy gains as a result of training. 

As noted in the federal regulations, mandated training frequency varies across team composition. While additional 

research is needed to confirm these findings, these results suggest that teams may benefit from time spent together 

outside of training with increases in confidence in their team’s ability to accomplish team tasks and incorporate new 

experiences into the team dynamic. Finally, this study provides an instrument development foundation upon which 

other emergency team training evaluation efforts can be based. Future research efforts may consider other ways to 

conceptualize the Big 5 and Coordinating Mechanisms and/or measure the differential impact of various efficacies on 

various team tasks. 

NIOSH OMSHR plans to develop more portable modalities (e.g., desktop applications, oculus rift, etc.) that may help 

to mitigate the accessibility issues leading to higher sample sizes, more in-depth analysis, and opportunities for 

longitudinal analysis. In addition to the benefits these technologies bring to research, increased accessibility can create 

opportunities for teams to interact more frequently, thereby strengthening the team processes believed to be so critical 

to successful team performance. 
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 10 of 11

REFERENCES 

Amico, G. Vincent (2009). Preface. In D. Schmorrow, J. Cohn, and D. Nicholson (Eds.), The PSI handbook of virtual 

environments for training and education: Developments for the military and beyond. Westport, CN: Praeger

Security International. 

Bauerle, T. J., & Mallett, L. G. (2013, May). Safety climate applied to crisis scenarios: Development of a measure of 

mine emergency preparedness climate. Presentation at the biennial Work, Stress and Health Conference, Los 

Angeles, CA.

Burke, M. J., Holman, D., Birdi, K. (2006). A walk on the safe side: The implications of learning theory for developing 

effective safety and health training. In G. P. Hodgkinson & J. K. Ford (Eds.), International review of 

industrial and organizational psychology (pp. 1–44). Chichester, England: Wiley. 

Cannon-Bowers, J. A., & Salas, E. E. (1998). Making decisions under stress: Implications for individual and team 

training. Washington, DC: American Psychological Association. Collins, C. G., & Parker, S. K. (2010). Team capability beliefs over time: Distinguishing between team potency, team 

outcome efficacy, and team process efficacy. Journal of Occupational and Organizational Psychology,

83(4), 1003-1023.

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor 

analysis in psychological research. Psychological Methods, 4(3), 272.

Ford, J. K., & Schmidt, A. M. (2000). Emergency response training: strategies for enhancing real-world 

performance. Journal of hazardous materials, 75(2), 195-215.

Galvin, J. M. (2008). Review of best practices for escape and rescue from underground coal mines in Australia. Galvin 

and Associates Pty Ltd, St Ives NSW, Australia: CDC contract no. 200-2008-M-24524 for NIOSH. 

Gruenfeld, D. H., Mannix, E. A., Williams, K. Y., & Neale, M. A. (1996). Group composition and decision making: 

How member familiarity and information distribution affect process and performance. Organizational 

Behavior and Human Decision Processes, 67, 1-15.

Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of 

Management, 21(5), 967-988.

Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational 

Research Methods, 1(1), 104-121.

Holton, E. F. (1996). The flawed four‐level evaluation model. Human Resource Development Quarterly, 7(1), 5-21.

Holton, E. F. (2005). Holton's evaluation model: New evidence and construct elaborations. Advances in Developing 

Human Resources, 7(1), 37-54.

Huckman, R. S., & Staats, B. R. (2011). Fluid tasks and fluid teams: The impact of diversity in experience and team 

familiarity on team performance. Manufacturing & Service Operations Management, 13(3), 310-328.

Linehan, C., Lawson, S., Doughty, M., & Kirman, B. (2009). Thereʼs no ʻIʼ in ʻEmergency Management Team': 

Designing and evaluating a serious game for training emergency managers in group decision making skills.

In Proceedings of the 39th Conference of the Society for the Advancement of Games & Simulations in 

Education and Training (pp. 20-27). Innovation North-Leeds Metropolitan University.

Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team 

processes. Academy of Management Review, 26(3), 356-376.

Prensky, M. (2001). Digital game-based learning. New York: McGraw Hill.

Robson, L. S., Stephenson, C.M., Schulte, P.A., Amick III, B.C., Irvin, E. L., Eggerth, D. E., Grubb, P. L. (2012). A 

systematic review of the effectiveness of occupational health and safety training. Scandinavian Journal of 

Work, Environment & Health, 38(3). 

Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of 

Psychology, 52(1), 471-499.

Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a “Big Five” in teamwork? Small Group Research, 36(5), 555-

599.

Schaafstal, A. M., Johnston, J. H., & Oser, R. L. (2001). Training teams for emergency management. Computers in 

Human Behavior, 17, 615-626.

Serfaty, D., Entin, E. E., & Volpe, C. (1993). Adaptation to stress in team decision-making and coordination. 

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 37, 1228-1232.
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2015

2015 Paper No. 15119 Page 11 of 11

Smith-Jentsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-related expertise in complex 

environments. In J. A. Cannon-Bowers & E. E. Salas (Eds.), Making decisions under stress: Implications for 

individual and team training (61-87). Washington, DC: American Psychological Association.

Smith-Jentsch, K. A., Kraiger, K., Cannon-Bowers, J. A., & Salas, E. (2009). Do familiar teammates request and 

accept more backup? Transactive memory in air traffic control. Human Factors, 51, 181-192.

Waller, M., Lei, Z., & Pratten, R. (2014). Focusing on Teams in Crisis Management Education: An Integration and 

Simulation-Based Approach. Academy of Management Learning & Education, 13, 208-221.