DOT/FRA/ORD-23/05 Final Report | February 2023 

Evaluation of Scenario-Based Training for Railroad 

Transportation Employees

i 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data 

sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other 

aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information 

Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision 

of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE

Technical Report

3. DATES COVERED (From - To)

May 2019 – September 2022

4. TITLE AND SUBTITLE

Evaluation of Classroom Centered Scenario-based Training for Railroad 

Transportation Employees

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S)

Jared Young, ORC ID 0000-0001-9502-3632

Jordan Multer, ORC ID 0000-0002-7818-110X

5d. PROJECT NUMBER

RR04AB/VK297

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

Volpe National Transportation Systems Center

55 Broadway 

Cambridge, MA 02142

8. PERFORMING ORGANIZATION

REPORT NUMBER

DOT-VNTSC-FRA-22-03

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

U.S. Department of Transportation

Federal Railroad Administration

Office of Railroad Policy and Development

Office of Research, Development and Technology

Washington, DC 20590

10. SPONSOR/MONITOR'S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT

NUMBER(S)

DOT/FRA/ORD-23/05

12. DISTRIBUTION/AVAILABILITY STATEMENT

This document is available to the public through the FRA website. 

13. SUPPLEMENTARY NOTES

COR: Michael Jones

14. ABSTRACT

In 2015, a peer review team (PRT) comprised of labor, craft employees, and railroad managers developed railroad scenario-based 

training (SBT). The PRT was responsible for reviewing close call reports and developing recommendations to address unsafe conditions 

on the railroad. The objective of SBT is to improve the coordination of teams to operate safely through exposure to the hazardous 

conditions that can lead to unsafe outcomes that railroads previously identified through observations, close call reports, incident reports, 

and accidents. The research evaluated the effectiveness of SBT to learn if the training is contributing to more effective learning of safetycritical behaviors. Among the three railroads that were able to conduct training sessions, each demonstrated that SBT can lead to slight 

improvements in students’ correct responses on training evaluation forms. While the differences in pre- and post-evaluation forms were

not significantly different for experienced employees, new employees displayed just under 17 percent improvement, which was 

statistically significant. This significant difference shows that SBT programs have the potential to teach new students about real world 

scenarios that they might not otherwise have been able to learn during their standard training.

15. SUBJECT TERMS

Knowledge acquisition, peer review team, PRT, scenario-based training, SBT, safety, teams, team training, learning, human factors

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF

ABSTRACT

18. NUMBER

OF

PAGES

30

19a. NAME OF RESPONSIBLE PERSON

Jared Young a. REPORT

UNC

b. ABSTRACT

UNC

c. THIS PAGE

UNC

19b. TELEPHONE NUMBER (Include area code)

(617) 494-2629

Standard Form 298 (Rev. 8/98)

Prescribed by ANSI Std. Z39.18

February 10, 2023
ii

METRIC/ENGLISH CONVERSION FACTORS

ENGLISH TO METRIC METRIC TO ENGLISH

LENGTH (APPROXIMATE) LENGTH (APPROXIMATE)

1 inch (in) = 2.5 centimeters (cm) 1 millimeter (mm) = 0.04 inch (in)

1 foot (ft) = 30 centimeters (cm) 1 centimeter (cm) = 0.4 inch (in)

1 yard (yd) = 0.9 meter (m) 1 meter (m) = 3.3 feet (ft)

1 mile (mi) = 1.6 kilometers (km) 1 meter (m) = 1.1 yards (yd)

1 kilometer (km) = 0.6 mile (mi)

AREA (APPROXIMATE) AREA (APPROXIMATE)

1 square inch (sq in, in2) = 6.5 square centimeters (cm2) 1 square centimeter 

(cm2

) 

= 0.16 square inch (sq in, in2

) 

1 square foot (sq ft, ft2) = 0.09 square meter (m2) 1 square meter (m2) = 1.2 square yards (sq yd, yd2) 

1 square yard (sq yd, yd2) = 0.8 square meter (m2) 1 square kilometer (km2) = 0.4 square mile (sq mi, mi2) 

1 square mile (sq mi, mi2) = 2.6 square kilometers (km2) 10,000 square meters 

(m2

) 

= 1 hectare (ha) = 2.5 acres

1 acre = 0.4 hectare (he) = 4,000 square meters (m2) 

MASS - WEIGHT (APPROXIMATE) MASS - WEIGHT (APPROXIMATE)

1 ounce (oz) = 28 grams (gm) 1 gram (gm) = 0.036 ounce (oz)

1 pound (lb) = 0.45 kilogram (kg) 1 kilogram (kg) = 2.2 pounds (lb)

1 short ton = 2,000 pounds (lb) = 0.9 tonne (t) 1 tonne (t) = 

= 

1,000 kilograms (kg)

1.1 short tons

VOLUME (APPROXIMATE) VOLUME (APPROXIMATE)

1 teaspoon (tsp) = 5 milliliters (ml) 1 milliliter (ml) = 0.03 fluid ounce (fl oz)

1 tablespoon (tbsp) = 15 milliliters (ml) 1 liter (l) = 2.1 pints (pt)

1 fluid ounce (fl oz) = 30 milliliters (ml) 1 liter (l) = 1.06 quarts (qt)

1 cup (c) = 0.24 liter (l) 1 liter (l) = 0.26 gallon (gal)

1 pint (pt) = 0.47 liter (l)

1 quart (qt) = 0.96 liter (l)

1 gallon (gal) = 3.8 liters (l)

1 cubic foot (cu ft, ft3) = 0.03 cubic meter (m3) 1 cubic meter (m3) = 36 cubic feet (cu ft, ft3) 

1 cubic yard (cu yd, yd3) = 0.76 cubic meter (m3) 1 cubic meter (m3) = 1.3 cubic yards (cu yd, yd3)

TEMPERATURE (EXACT) TEMPERATURE (EXACT)

[(x-32)(5/9)] °F = y °C [(9/5) y + 32] °C = x °F

For more exact and or other conversion factors, see NIST Miscellaneous Publication 286, Units of Weights and 

Measures. Price $2.50 SD Catalog No. C13 10286 Updated 6/17/98
iii

Acknowledgements

The authors would like to thank the Federal Railroad Administration (FRA) Region 1 office for 

the assistance in gathering railroads to participate in a study of an experimental training program. 

A special thank you to Edward Flynn at FRA for his continued involvement in pushing this effort 

forward. Finally, thank you to the participating railroads for sharing vital data needed for the 

analysis of this study.
iv

Contents

Executive Summary ................................................................................................................... 1

1. Introduction ............................................................................................................. 2

1.1 Background ..............................................................................................................2

1.2 Objectives................................................................................................................2

1.3 Overall Approach.....................................................................................................2

1.4 Scope .......................................................................................................................2

1.5 Organization of the Report .......................................................................................3

2. How SBT Works..................................................................................................... 4

2.1 Background ..............................................................................................................4

2.2 Design......................................................................................................................5

2.3 Training Objectives..................................................................................................5

2.4 Current SBT Programs.............................................................................................6

3. Method ...................................................................................................................10

3.1 Evaluation Purpose................................................................................................. 10

3.2 Evaluation Questions.............................................................................................. 10

3.3 Constraints and Limitations.................................................................................... 10

3.4 Overview................................................................................................................ 10

3.5 Training Conditions................................................................................................ 11

3.6 Participants............................................................................................................. 11

3.7 Procedure ............................................................................................................... 12

4. Results ...................................................................................................................14

4.1 Railroad 1............................................................................................................... 14

4.2 Railroad 2............................................................................................................... 15

4.3 Railroad 3............................................................................................................... 16

5. Conclusion .............................................................................................................18

6. References..............................................................................................................19

Appendix A: Railroad 1 Conditions for Learning Questions and Responses..............................20

Appendix B: Railroad 2 Conditions for Learning Questions and Responses ..............................21
v 

Illustrations

Figure 1. Room Layout for Railroad 1 SBT Class........................................................................6

Figure 2. Room Layout for Railroad 2 SBT Class; Left: Students and Observers Room Right: 

Dispatcher and Trainmaster Room.......................................................................................7

Figure 3. Room Layout for Railroad 3 SBT Class; Left: Students and Observers Room Right: 

Dispatcher and Trainmaster Room.......................................................................................8

Figure 4. Room Layout for Railroad 4 SBT Class........................................................................9

Figure 5. Questions and Responses for COL Assessment: Railroad 1 ........................................ 15

Figure 6. Questions and Responses for COL Assessment: Railroad 2 ........................................ 16
vi 

Tables

Table 1. Training Conditions by Railroad.................................................................................. 11

Table 2. Overview of Scenario Based Training Programs.......................................................... 12

Table 3. Railroad 1 Pre- and Post-Assessment Results............................................................... 14

Table 4. Railroad 2 Pre- and Post-Assessment Results............................................................... 16

Table 5. Railroad 3 Pre- and Post-Assessment Results............................................................... 17
1 

Executive Summary

To work safely and efficiently, railroad employees acquire knowledge and skills needed to 

perform their jobs through a combination of formal classroom training and on-the-job mentoring 

by experienced practitioners in the same craft. Through railroad participation, scenario-based 

training (SBT) was developed from the Federal Railroad Administration (FRA)’s Confidential 

Close Call Reporting System (C3RS). In an effort to understand how SBT impacts learning, FRA 

reached out to the Volpe National Transportation Systems Center to evaluate the current efforts 

by several railroads in implementing SBT. The peer review team (PRT), comprised of labor craft 

employees and railroad managers, is the group responsible for reviewing event reports and 

recommending corrective actions. The PRT advised that creating training based on some of the 

scenarios documented in the reports could be one way to address unsafe conditions on the 

railroad.

In response to a variety of close call events, the PRT recommended developing training 

involving the use of role playing exercises where the crew or individual could work through the 

unsafe event and learn how to respond safely. In 2015, Railroad 4 in this study experimented 

with teaching new students how to handle scenarios when the railroad received multiple close 

call reports and observed challenges during operational testing and other safety related incidents. 

This training came to be known as SBT, which addressed a specific set of conditions that can 

lead to harm.

One objective of SBT was to improve the coordination of teams to operate safely through 

exposure to the hazardous conditions that can lead to unsafe outcomes that railroads previously 

identified through observations, close call reports, incident reports, and accidents. A second 

objective was to teach employees how to safely operate in a specific hazardous condition that 

can result in an unsafe outcome.

The current research evaluated the effectiveness of SBT to assess whether the training is 

contributing to more effective learning of safety critical behaviors. The study used a set of 

questions to measure knowledge before and after training. Four railroads participated in the 

study. To keep the identify of these railroads confidential, they will be referred to as Railroad 1, 

Railroad 2, Railroad 3, and Railroad 4. Railroads 1 and 2 used the SBT for refresher training for 

experienced employees (i.e., block training) and Railroad 3 used SBT with new student 

employees. Railroad 4 was unable to participate during the study period due to COVID-19. 

Among the three railroads that were able to conduct training sessions, each demonstrated that 

SBT can lead to improvements in knowledge acquisition. New employees showed greater 

improvements compared to experienced employees. Knowledge acquisition improved by 17

percent for new employees and 1–3 percent for experienced employees. This improvement 

shows that SBT has the potential to teach new students about real world scenarios that they 

might not otherwise encounter during their standard training.
2 

1. Introduction

The Federal Railroad Administration (FRA) reached out to the Volpe National Transportation 

Systems Center (Volpe Center) to evaluate the current efforts by several railroads in 

implementing scenario-based training (SBT), a form of training that several passenger railroads 

currently use. The research team examined how several passenger railroads incorporated SBT 

into their new employee training and recurrent training (i.e., annual refresher training or block 

training) for locomotive engineers, conductors, dispatchers, and block operators and measured 

the impact of this training on classroom learning. 

1.1 Background

To work safely and efficiently, new railroad employees acquire the knowledge and skills needed 

to perform their jobs through a combination of formal classroom training and on-the-job 

mentoring by experienced practitioners in the same craft. For locomotive engineers, it can take 2 

years before they earn their certification. For conductors, the training can range from 8 to 12 

months. 

SBT has proven effective in several other domains, including automobile operations. Dating 

back to 2003, research on automotive SBT by McKnight and McKnight concluded from a 

detailed analysis of crash reports that the overwhelming majority of motor vehicle crashes 

involving novice drivers (i.e., 16–19 years old) resulted from their failure to employ safe 

operating practices and to recognize or anticipate dangers (McKnight and McKnight, 2003).

A passenger railroad developed SBT in response to event reports received from FRA’s

Confidential Close Call Reporting System (C3

RS). In response to some event reports, a peer 

review team (PRT) identified training as corrective action to some of the factors contributing to 

these events. The training addressed many of the types of events reported to C3RS and involved 

role playing so students could experience these events in a safe environment where they could 

make mistakes and learn from these experiences. FRA sought a wider demonstration and 

evaluation of this training to learn whether it would be effective in promoting learning and to 

share this information more broadly with industry stakeholders.

1.2 Objectives

The purpose of this evaluation was to evaluate the impact of SBT training on railroad employees 

in the classroom.

1.3 Overall Approach

The research team observed the training at each of the participating railroads to document how 

the railroads implemented SBT. Following the observations, the team created three surveys for 

evaluating the impact of the training on student learning. The participating railroads administered 

the three surveys and researchers reviewed and documented the results.

1.4 Scope

The evaluation focused on the impact of the training on classroom learning and did not evaluate 

the impact on application of student learning in railroad operations nor did it measure the impact 

on safety outcomes.
3 

1.5 Organization of the Report

Section 2 describes SBT and how the railroads decided to explore this form of training. Section 3

details the methods for evaluating the impact of SBT classroom training and Section 4 provides 

the results of the evaluation. Section 5 offers conclusions on the work conducted. Appendix A

contains the questions provided for Railroad 1 and Appendix B lists the questions given to 

Railroad 2.
4 

2. How SBT Works

This section provides background on SBT, how it is designed, and how railroads currently 

implement this training.

2.1 Background

To work safely and efficiently, new railroad employees acquire the knowledge and skills needed 

to perform their jobs through a combination of formal classroom training and on-the-job 

mentoring by experienced practitioners in the same craft. This employee training focuses on 

learning the language of railroad operations, the operating rules, how to operate the equipment 

for which they are responsible, and the characteristics of the territory they will operate. The 

duration of this training varies by craft. For locomotive engineers, it can take up to 2 years before 

they earn their certification. For conductors, the training can range from 8 to 12 months. 

Recurrent or refresher training, which occurs on a periodic basis (e.g., annually) in a classroom 

environment, enables the railroads to reinforce or emphasize safety issues of concern to the 

railroad. Some safety topics are repeated year after year, while some safety topics vary each year 

depending upon the safety concerns that arose during the previous year. This training varies in 

duration depending upon the railroad. For both new employee training and refresher training, 

employees receive instruction according to their craft.

SBT has proven effective in several domains, including automobile operations. Dating back to 

2003, research on automotive SBT by McKnight and McKnight concluded from a detailed 

analysis of crash reports that the overwhelming majority of motor vehicle crashes involving 

novice drivers (16–19 years old) resulted from their failure to employ safe operating practices 

and a failure to recognize or anticipate dangers (McKnight and McKnight, 2003). Prior to this 

research by McKnight and McKnight, it was assumed that novice drivers were overrepresented

in vehicle crashes because they were deliberately driving in a more reckless manner. However, 

this study found that novice drivers were unaware that their behavior was dangerous, and they 

could become safer drivers with more education. Since 2006, computer-based scenario training 

programs have been proven effective at reducing behaviors that lead to automotive crashes. 

These programs include Risk Awareness and Perception Training [RAPT] (Pollatsek et al., 

2006); Anticipate, Control, and Terminate [ACT] (Muttart, 2013); Secondary Task Regulatory & 

Anticipatory Program [STRAP] (Krishnan et al. 2015); Forward Concentration and Attention

Learning [FOCAL] (Divekar et al., 2013); Less Aggressive Goals [LAG] (Zhang, Romoser & 

Fisher, 2014); and Accelerated Curriculum to Create Effective Learning [ACCEL] (Fisher, 

Young, Zhang, Knodler, Samuel, 2017). Similar training in the rail industry might also prove 

effective. 

In the railroad domain, two FRA studies, one on stop signal overruns and another on running 

through misaligned switches, made clear that training and experience play a vital role in the safe 

and efficient operations of train handling (Multer, Safar, & Roth, 2019; Safar et al., 2019). In 

these studies, the authors recommended the adoption of SBT as a method for reducing the 

likelihood of stop signal overruns.

SBT for railroads was developed by a passenger railroad in response to event reports received 

from C3

RS. The PRT identified training as corrective action to some of the factors contributing 

to these events. The training addressed many of the types of events reported to C3RS and 

involved role playing so students could experience these events in a safe environment where they
5 

could make mistakes and learn from their experiences. After observing this training, FRA sought 

a wider demonstration and evaluation to learn whether this training was effective in promoting 

learning and share this information with industry stakeholders.

In 2015, the railroad that developed a SBT program experimented with teaching new students 

how to handle scenarios for which the railroad received multiple close call reports, observed 

challenges during operational testing, and safety related incidents. This training addressed a 

specific set of conditions that can lead to harm associated with railroad operations. 

In 2019, FRA Region 1 held a meeting to promote the use of SBT. Several passenger railroads 

attended this meeting to learn about the training method and its potential benefits. Following this 

meeting, three railroads expressed interest and began to develop their own programs modeled 

after the SBT program. Each railroad adapted the training to meet its safety concerns. 

2.2 Design

Training that is scenario-based focuses on a specific set of conditions that can lead to harm 

associated with railroad operations. The training addresses the behaviors that teams can use to 

prevent these conditions from causing harm or to mitigate the consequences of the harm. This 

can include teams that work closely together in time and space or teams that may be separated 

geographically or by time.

SBT uses methods like role playing, human-in-the loop simulation, and controlled field settings 

that enable teams to experience the conditions in which the harm may occur. Teams learn to 

recognize hazardous situations when they arise and to practice behaviors and communications 

that lead to safe outcomes or, if an unsafe event occurs, to mitigate the consequences.

SBT for new and experienced employees was designed to comply with Federal Title 49 Code of 

Regulations (CFR) Parts 240 and 242. The training involves participants from all crafts that 

would interact during the envisioned scenario. For example, if the scenario involved train 

operations, the exercise could include train crew members, a locomotive engineer, a conductor, 

an assistant conductor (brakeman), and a dispatcher. Depending upon the availability of 

employees from the different crafts and the design of the training scenario, railroad managers or 

training staff may role-play as one of the necessary craft employees.

2.3 Training Objectives

For the railroads, one objective of SBT is to improve team coordination and communication, 

which allows the team to operate safely through exposure to the hazardous conditions that can 

lead to unsafe outcomes. A second objective is to teach employees how to safely operate in a 

specific hazardous condition that could result in a safety incident.

The training places the student1 in a controlled, safe environment (i.e., classroom or field) where 

they can experience the same type of real-world unsafe conditions that they might encounter in 

the field. The trainee learns how to recognize unsafe conditions, act to avoid unwanted events,

and safely operate the train. Students may play one of the roles in the scenario or observe the 

scenario as it plays out.

1 A student refers to either a new employee that is in the process of becoming certified to operate under Part 240 or 

242 or an experienced certified employee that is receiving periodic refresher training.
6 

2.4 Current SBT Programs

Four passenger railroads volunteered to participate in this study and are represented as Railroad 

1, Railroad 2, Railroad 3, and Railroad 4. Each developed a training program adapted to their 

railroad and each organization implemented it in different ways. Railroad 1 conducted their first 

group SBT session in the fall of 2018. Railroad 2 began conducting individual craft (non-group) 

SBT in early 2018. Railroad 3 conducted their first training session in 2016 and Railroad 4 held 

their first session in November 2019. The research team studied one instance of SBT training 

from each railroad. The following section describes how each passenger railroad conducted that 

instance of student training using SBT. 

2.4.1 Railroad 1

Railroad 1 offered SBT as a 1-hour module during their annual block (i.e., refresher) training 

course. All employees participated in this training. During this training, the SBT module 

consisted of two scenarios drawn from issues identified in C3

RS, operations testing, or incidents 

that took place during the previous year. Students participated in scenarios either around a table 

in the classroom (see Figure 1 for room layout), in a human in-the-loop simulator, or in the field 

(i.e., on actual trains or in a yard). Once the scenario was completed, feedback was provided by 

instructors and other students in the class.

Figure 1. Room Layout for Railroad 1 SBT Class

2.4.2 Railroad 2

Training took place in a large room with chairs and desks arranged in a U-shaped layout. In the 

center of the U were two tables and chairs, one for a locomotive engineer and another for a 

conductor (Figure 2). This was the railroad’s first training session with experienced employees, 

having previously conducted a SBT session with new employees in November 2019. Before the 

training session began, the instructors described SBT so the students understood the purpose of 

the training. Railroad 2 designed 14 scenarios for the students to experience. The students acting 

out each scenario sat at the two tables in the center of the U. They were given radios, paper 

copies of bulletin orders, rules forms, and a timetable. At the start of each scenario, the students 

were given an envelope that described the training situation. As the students worked through the 
7 

exercise, experienced trainmasters and dispatchers played their roles in another room, hidden 

from the participants. At the conclusion of each exercise, the instructors and observers discussed 

the scenario with the students and offered feedback on how they could improve their 

performance.

Figure 2. Room Layout for Railroad 2 SBT Class; Left: Students and Observers Room 

Right: Dispatcher and Trainmaster Room

2.4.3 Railroad 3

Railroad 3 offered SBT to new student engineers, conductors, dispatchers, and block operators 

prior to certification. Their training took place in a room with chairs and desks arranged in a Ushaped layout (Figure 3). In the center of the U were chairs for participants acting as passengers. 

A table for an engineer was in the front of the U-shaped layout facing a projector screen. Another 

table for a conductor was behind the chairs in the center of the U, with the chair for the student 

either facing in the opposite direction of the screen (i.e., looking away) or facing the screen, 

depending on the scenario. For example, the engineer’s chair faced away from the screen during 

reverse train movements to simulate how the engineer would not be able to see out of the 

window while performing a reverse train movement, and then faced the screen when the train 

was making a forward movement. Road foremen, union representatives, and other managers sat 

around the table. The observers provided feedback to the students following the completion of 

each scenario.

The instructors introduced the training and described how it was developed. Then the instructor 

invited two students up to begin the first of six scenarios. The students were given a piece of 

paper describing the situation they would face and their role in the scenario. The two students 

performed a job briefing to introduce their roles to each other and then took their seats. If the 

scene involved video showing movement of the train, the video would begin when the engineer 

indicated that he was beginning operation of the train. The instructor at the front of the room 

controlled the video using Microsoft video playback software, and would start and stop the video 

at the direction of the student engineer. The conductor and engineer would communicate as 

needed during the scene. The students both had radios to communicate with the tower/block 
8 

operator or dispatcher located in a separate room. The training was recorded on video and 

students used microphones located on each table to support the video recording as well as enable 

the other students, trainers, union representatives, and guests to better hear them speaking. 

All the feedback following the scenario came from people sitting around the table (e.g., 

managers, union representatives, and trainers). No feedback or comments were solicited from the 

students sitting in the back of the room.

Figure 3. Room Layout for Railroad 3 SBT Class; Left: Students and Observers Room 

Right: Dispatcher and Trainmaster Room

2.4.4 Railroad 4

Railroad 4 offered training to new student locomotive engineers, conductors, and dispatchers 

toward the end of their training, prior to being certified to operate. The training took place over 3 

days and consisted of 36 role playing scenarios. Students observing the exercises sat around a Ushaped conference table (Figure 4), and the students participating in the exercise sat at individual 

tables in the middle of the U-shaped table. Each scenario involved an engineer, a conductor, and 

a dispatcher.

Over the course of the 3 day training, each student participated in at least one scenario. Each 

scenario began with the participants receiving index cards describing the situation (e.g., 

performing a reverse train movement) and the role they would play. The crew worked together to 

achieve the goal of the scenario, with a focus on how to perform the work safely. During each 

exercise, one student received a “wildcard” that created an additional challenge that the team 

would need to address. About 95 percent of the time, teams were able to work through a scenario 

to its completion. In a small portion of the exercises, the teams were asked to end their scenario 

because they adopted incorrect procedures. At the conclusion of each scenario, the instructors 

went around the room and asked for feedback from students and observers. Instructors for each 

of the three crafts also provided feedback to the students. Following this feedback, the instructors 

reviewed the real-world event and explained in detail what occurred. They also gave feedback to 

the students on how to handle the scenarios successfully. 
9 

Figure 4. Room Layout for Railroad 4 SBT Class 
10 

3. Method

This section describes the method researchers used for evaluating the impact of SBT on 

employee learning. The team collected data over a period of 12 months, from September 2021 to 

August 2022. 

3.1 Evaluation Purpose

Researchers evaluated the effectiveness of SBT to learn if the training contributed to more 

effective learning of safety critical behaviors. Specifically, the team evaluated how students 

responded to the training and whether students demonstrated learning. Researchers posed three

questions regarding the effectiveness of the training, as discussed below. 

3.2 Evaluation Questions

The questions used in the training evaluation were: 

1. Do individuals demonstrate knowledge of what actions to take in response to the 

scenarios?

2. How does training effectiveness vary with use of role playing alone compared to role 

playing using video or computer-based simulation or OJT in the field?

3. To what extent does the training focus on individual knowledge and skills vs. team-based 

knowledge and skills?

3.3 Constraints and Limitations

The training evaluation was intended to help the railroads and FRA learn whether SBT achieves 

improvements in learning behaviors and strategies for safe job performance. Researchers also 

sought to identify strengths and weaknesses in the different ways the training was implemented.

For this research, the team depended upon the railroads engaged in SBT to collect the data 

needed for evaluation. This includes data collection prior to, during, and after training. In several 

cases, the railroads were unable to obtain evaluation data due to time limitations during the 

training. 

Due to the impacts of COVID-19, the participating railroads focused their attention on critical 

aspects of operation and in-person training was minimized to include only that deemed 

necessary. For this reason, Railroad 4 was not able to conduct any training during the research 

period and therefore was not able to submit training data for evaluation. Also, while railroads 

were focused on current operations, requests for information from researchers was sometimes 

delayed. These delays contributed to a delay in completing this study.

3.4 Overview

The following method was used to evaluate this training: 

1. Compare knowledge retained by pre-training and post-training instruments.

a. The study identified the following learning objectives:
11 

i. What knowledge should students demonstrate? How much is scenariospecific?

ii. What skills and knowledge should be generalized (e.g., communications 

skills)?

b. The study assessed whether learning took place. 

i. How well did students learn the information defined by the learning

objectives? 

ii. How did learning vary with the methods used (e.g., classroom, video, 

computer-based simulation)? 

2. Administer an engagement questionnaire to measure how the students reacted to the 

conditions for learning.

a. Did the students who participated in the training find it helpful?

b. Did the students believe that they understood the goals of the training?

c. Did the students feel engaged in the conversation?

3.5 Training Conditions

Table 1 shows the conditions under which training was conducted and the student crafts that 

participated in the training. The conditions reflect differences in the number of scenarios that the 

railroads presented during training and the media used.

Table 1. Training Conditions by Railroad

Railroad 1 2 3 4

Duration 1 Hour 2–3 Hours 1 Day 3 Days

Media 

Used

Tabletop Role 

Play uses paper 

and radios

Role Playing with Video 

of moving train uses 

paper, radios, and 

computer-based video

Field, Human in the Loop 

Simulators, or Tabletop Role 

Play uses paper, radio, computer 

based simulation, and equipment 

in the field

Tabletop Role 

Play uses paper 

and radios

Crafts 

Involved 

Engineer, 

Conductor, 

Dispatcher

Engineer, Conductor, 

Dispatcher, Foreman

All Crafts

Engineer, 

Conductor, 

Dispatcher

3.6 Participants

Four passenger railroads volunteered to participate in this study. To provide confidentiality, they 

are referred to as Railroad 1, Railroad 2, Railroad 3, and Railroad 4.

The SBT training for each of the participating railroads varied in duration; Table 1 shows the 

number of scenarios in which the students participated. The training duration for the participating 

railroads was between 1–3 days depending upon the number of scenarios designed by each 

railroad.

The craft of the students that participated in SBT also differed between railroads. Table 2 shows 

the number of employees participating by craft and railroad. Only Railroad 3 included new 

employees learning their craft; the participants from Railroads 1 and 2 were experienced 
12 

employees. This research focused on the self-assessments of locomotive engineers, conductors, 

and dispatchers and/or block operators. Participation by individual students occurred in one of 

two ways, either participating in or observing the scenarios. Railroad instructors selected the 

students to participate in each scenario. While most scenarios involved a locomotive engineer 

and conductor, some also included student dispatchers and block operators. The number of 

students and scenarios determined the number of scenarios in which students participated. Since 

only two students generally participated in each scenario, students observed more scenarios than 

they participated in. Instructors and non-student employees also participated in the scenarios by 

playing the other roles required in some of the scenarios.

Table 2. Overview of Scenario Based Training Programs

Railroad 1 2 3

Number of 

students

Conductor – 76

Assistant Conductor – 18

Locomotive Engineer – 37

Total – 131

Conductor – 15

Locomotive Engineer – 5

Total – 20

Conductor – 7

Locomotive Engineer – 6

Block Operator – 1

Total – 14

Training Type Block Training Block Training New Student Training

Duration 1 hour 2 hours 3 days

3.7 Procedure

Researchers administered three surveys to evaluate the impact of training on learning. 

1. Pre-training, instructor-administered assessment of knowledge and skill acquisition

2. Post-training, instructor-administered assessment of knowledge and skill acquisition

3. Assessment of conditions for learning (COL)

3.7.1 Pre- and Post-Training Instructor-Administered Assessment

The research team developed a paper-based assessment. Each railroad’s training instructors 

administered the assessment before SBT began and at the conclusion of the training. The 

assessment measured each student’s knowledge in responding to the situations encountered in 

the training scenarios. 

The research team worked with the training instructors at each of the participating railroads to 

develop the questions for each assessment. The content for the questions came from the learning 

objectives developed by the railroads before the training began and consisted of multiple-choice

questions. The post-assessment version of the instructor-administered survey distinguished

responses from students that participated in the scenario from students that observed the training. 

This design enabled discrimination in learning impact between observations of the scenario 

compared to participation in the scenario.

3.7.2 Questionnaire Development to Assess the Conditions for Learning

After employees completed the training, instructors administered a third and final questionnaire 

to assess how the conditions present during the training (e.g., media used, engagement with 

instructors, engagement with other students, etc.) contributed to learning and what conditions 

might have impaired learning. 
13 

3.7.3 Survey Administration

Researchers provided the three SBT surveys to the training staff and instructed them on how to 

administer the surveys in the classes. The pre-assessment survey was administered by the 

training instructors prior to the training taking place. Following the training, the instructors 

administered the post-assessment survey and the COL questionnaire. These forms were collected 

immediately after completion and given to the research team to analyze.

3.7.4 Data Collection and Analysis

The research team measured the change in learning between the assessments and scored the 

questionnaires for each railroad. Aggregate scores were summarized across the participating 

railroads.
14 

4. Results

The research team collected data from the surveys discussed in Section 3.7 (i.e., the pre- and 

post-training assessments and the COL questionnaire) from three of the participating railroads 

(i.e., Railroads 1, 2, and 3) over a 12 month period. Railroad 4 was unable to perform any 

training classes during this time due to restrictions from the COVID-19 pandemic.

Each railroad administered the surveys in the same fashion. The pre-training assessment was 

given before the training occurred and the post training assessment and the COL questionnaire

were completed after the training. As shown in Table 2, each railroad conducted their training in 

a slightly different way, so the team performed analysis individually for each railroad to 

determine if there was a statistical significance between the pre- and post-training assessment. In 

addition, researchers compiled summary statistics from the COL questionnaire to gain a better 

understanding of how effective the students believed this type of training might be for them.

4.1 Railroad 1

Railroad 1 conducted 11 classes, with an average class size of 11 students. SBT was offered to 

experienced employees during their yearly refresher training. The breakdown of craft is listed in 

Table 3. 

4.1.1 Pre- and Post-Training Assessment

Overall, students responses showed a 1.4 percent in learning from pre-test to post-test. To

determine if this change was statistically significant, the researchers performed a two-sample 

proportion test. The change in performance was not statistically significant. 

While the results were positive (i.e., the post-evaluation results were better than the preevaluation results), the change was small. The researchers also looked at how craft may affect 

the improvement from the pre-evaluation and post-evaluation questioners by using a Tukey HSD 

multiple comparisons of means test. This test displayed a p-value of 0.535 which once again was 

not statistically significant. However, results for each craft were positive and a small 

improvement was shown for each craft (Table 4).

Table 3. Railroad 1 Pre- and Post-Assessment Results

Craft Number of 

Students

Pre-Test 

Score (%)

Post Test 

Score (%)

Percent 

Change

Conductor 76 80 81 1 

Assistant 

Conductor 18 81 82 1 

Locomotive 

Engineer 37 74 77 3 

Overall 131 78 79 1 

4.1.2 COL Assessment

Railroad 1 collected the COL questionnaire responses following the post-training assessment. 

The COL for Railroad 1 consisted of 14 questions. Out of the 131 students whose responses were 

analyzed, each student participated in an average of 1.6 scenarios and observed an average of 
15 

1.96 scenarios. Questions 8 through 13 were rated on a scale from 1 to 4. The average response 

was 3.85, indicating that overall students felt this training was very useful. The lowest rating was 

given for question 17, which asked students how challenging they felt the training was; on 

average the students rated the course difficulty at 1.9 out of 4, meaning it was not very 

challenging for the students. A select list of questions and responses can be found in Figure 5, 

and the complete list of questions and responses can be found in Appendix A. 

Figure 5. Questions and Responses for COL Assessment: Railroad 1

4.2 Railroad 2

The team evaluated one class conducted by Railroad 2 over the research period. Approximately 2 

hours of SBT was given to experienced employees during their yearly refresher training. The 

breakdown of craft is listed in Table 4. Researchers analyzed the responses from 20 students and 

found an improvement of 3 percent in evaluation performance scores.

4.2.1 Pre- and Post-Training Assessment

To determine if this change was statistically significant, the researchers performed a two-sample 

proportion test. This test produced a p-value of 0.3632. Using an a = .05, the team concluded that 

because the p-value is greater than a, the difference between the proportion for the preevaluation questionnaire is no different than the proportion for the post-evaluation questionnaire. 

This means that there is no significant difference between pre-scenario evaluations and postscenario evaluations. While the results were positive (i.e., the post-evaluation results were better 

than the pre-evaluation results), the difference was not great enough to show a statistical 

significance given the relatively small sample size. 

The researchers also looked at how craft may affect improvement from the pre-evaluation and 

post-evaluation surveys. Researchers then used a Tukey HSD multiple comparisons of means

test. This test displayed a p-value of 0.0056. Since the p-value was less than a = .05, researchers

concluded that there is a significant difference between how the crafts preformed on the 

assessments, with greater improvements shown from the conductors. 
16 

Table 4. Railroad 2 Pre- and Post-Assessment Results

Craft Number of 

Student

Pre-Test 

Score (%)

Post Test 

Score (%) Percent Change

Conductor 15 67 70 3 

Locomotive 

Engineer 5 88 90 2 

Overall 20 72 75 3 

4.2.2 COL Assessment

The COL for Railroad 2 consisted of 15 questions. Out of the 20 students whose responses were 

analyzed, each student participated in an average of 3 scenarios and observed an average of 4.85 

scenarios. Questions 7 through 12 were rated on a scale from 1 to 4. The average response was 

3.81, indicating that overall students felt this training was very useful. Once again, the lowest 

rating was given for question 16, which asked students how challenging they felt the training 

was. On average the students rated the course 3.2 out of 5 on difficulty, meaning it was not very 

challenging for the students. A select list of questions and responses can be found in Figure 6 and 

the complete list of questions and responses can be found in Appendix B. 

Figure 6. Questions and Responses for COL Assessment: Railroad 2

4.3 Railroad 3

Researchers evaluated on class conducted by Railroad 3 during the research period. SBT was 

given to new student trainees at the end of their formal training and lasted 1 day. Table 5 lists the 

breakdown of craft. The team analyzed the responses from 14 students and observed an 

improvement of 16.67 percent in performance from pre- to post-evaluation. 

4.3.1 Pre- and Post-Training Assessment

To determine if this change was statistically significant, researchers performed a Chi-squared

test. The results were statistically significant (χ2 = 8.7231, df = 1, p = 0.003). Using an a = .05, 

researchers concluded that because the p-value was less than a, the hypothesis test shows that 

SBT demonstrated a statistically significant improvement in knowledge acquisition from the pretraining assessment to the post-training assessment.
17 

The researchers also looked at how craft may affect the improvement from the pre-evaluation 

and post-evaluation questioners. The team used a Tukey HSD multiple comparisons of means 

test to compare the improvements between the different crafts (F-value = 7.777, df = 2, p = 

0.0009). Since the p-value was less than a = .05, the team concluded that there is a significant 

difference by craft. While the conductors showed greater improvements than the locomotive 

engineers, the differences were not statistically significant, possibly due to the small sample size 

in each of these groups. All the crafts showed improvement from pre-assessment to postassessment except for the block operator craft. The block operators scored 100 percent on both 

the pre- and post-assessment, therefore there was no opportunity to demonstrate an improvement 

in learning.

Table 5. Railroad 3 Pre- and Post-Assessment Results

Craft Number of 

Student

Pre-Test 

Score (%)

Post Test 

Score (%)

Percent 

Change

Conductor 7 80 100 25 

Locomotive 

Engineer 6 90 100 11 

Block 

Operator 1 100 100 0 

Overall 14 86 100 17 

4.3.2 COL Assessment

Due to time constraints, Railroad 3 only asked three questions on their COL assessment. Out of 

the 14 students whose responses were analyzed, each student participated in an average of 1 

scenario and observed an average of 6.2 scenarios. Students rated the quality of the media used 

in training moderately high with an average rating of 3.8 out of 5. This indicates that the students 

found the media somewhat helpful.
18 

5. Conclusion

In this study, researchers found that SBT demonstrated greater improvements in learning for new 

student employees than for experienced employees. New student employees improved 

approximately 17 percent from pre-assessment to post-assessment. This improvement suggests 

that SBT has the potential to teach new students about real world scenarios that they might not 

otherwise encounter during their standard training. Replicating this research with more railroads, 

including freight service, and larger samples will inform whether these results generalize across 

the railroad industry.

For the two railroads that used SBT for refresher training, learning improved from the preassessment to the post-assessment by 1–3 percent. More research is needed to understand why 

the performance improvements for experienced employees were smaller than those for new 

employees and how to improve performance for this group. 

The three railroads that returned results tailored the SBT, the media used in training, and the 

three surveys to fit their timeframe and priorities. Railroad 1 used radios and paper to support a 

role playing activity while Railroad 2 included radios, paper, and video to support role playing. 

Railroad 3 used radios, paper, and computer-based simulation to support role playing activity. 

These differences make it difficult to compare results between railroads, although there were 

some similarities in the questions on the COL assessments. 

Responses comparing these COLs suggest that both new employees and experienced employees 

found the training useful. While the results of this study are promising, more research is needed 

to understand how to further improve on these findings, especially with experienced employees. 

For new students, replicating this research with larger sample sizes and variations in how 

railroads apply SBT is needed to better understand the effects of this training on learning. 

Two key aspects of this training were not evaluated by this study but deserve consideration. This 

training focused on events that each of the railroads experienced as either close calls or incidents. 

The training provided a mechanism to learn how to address these risks in a safe environment. 

Learning whether this training reduces the potential for these events to recur would be valuable 

but difficult to measure. A second aspect of the training addressed team interactions and focused 

on how employees can improve communication and collaboration with their teams. Like crew 

resource management and non-technical skills training, SBT can provide the soft skills needed 

for successful team performance within the railroad industry. Learning how this training affects 

team interactions in the field can inform railroad training methods for new and experienced 

employees.
19 

6. References

Cook, D. A., Thompson, W. G., Thomas, K. G., Thomas, M. R., & Pankratz, V. S. (2006). 

Impact of Self-Assessment Questions and Learning styles in Web-Based Learning: A 

randomized, Controlled Crossover Trial. IT in Medical Education, 231–238. 

Divekar, G., Pradhan, A. K., Masserang, K. M., Reagan, I., & Fisher, D. L. (2013). A simulator 

evaluation of the effects of attention maintenance training on glance distribution of 

younger novice drivers inside and outside the vehicle. Transportation Research Part F: 

Traffic Psychology and Behaviour, 20, 154–169. 

Fisher, D. L., Young, J., Zhang, L., Knodler, M., & Samuel, S. (2017). Accelerating Teen Driver 

Learning: Anywhere, Anytime Training. Washington, DC: AAA Foundation for Traffic 

Safety.

Krishnan, A., Samuel, A., Dundar, C., Romoser, M. R., & Fisher, D. L. (2015). Evaluation of a 

Hazard Anticipation Training Program (STRAP) on Secondary Task. Transportation 

Research Board 94th Annual Meeting (pp. 1–18). Washington, DC: Transportation 

Research Board.

McKnight, J. A., & Mcknight, S. A. (2003). Young novice drivers: careless or clueless? Accident 

Analysis and Prevention, 921–925. 

Multer, J., Safar , H., & Roth , E. (2019). Why do Passenger Trains Pass Stop Signals? A 

Systems View. Technical Report No. DOT/FRA/ORD-19/19, Washington, DC: Federal 

Railroad Administration.

Muttart, J. W. (2013). Identifying hazard mitigation behaviors that lead to differences in the 

crash risk between experienced and novice drivers. Doctoral Dissertations Available 

from Proquest. AAI3589108.

Perera, J., Perera, J., Abdullah, J., & Lee, N. (2009). Training simluated patients: evaluation of a 

training approach using self-assessment and peer/tutor feedback to improve performance. 

BMC Medical Education, 9. 

Pollatsek, A., Reichle, E. D., & Rayner, K. (2006). Tests of the E-Z Reader model: exploring the 

interface between cognition and eye-movement control. Cognitive Psychology, 52(1), 1–

56. 

Safar, H., Roth, E., Multer, J., & France, M. (2019). Why Do Passenger Trains Run through 

Switches in the Rail Yard? Technical Report No. DOT/FRA/ORD-19/37, Washington, 

DC: Federal Railroad Administraion.

Tronsmoen, T. (2008). Associations between self-assessment of driving ability, driver training 

and crash involvement amoung young drivers. Transportation Research Part F, 334–346. 

Wänglund, A. (2011). Assessing Subjective Driving Skills: An Example of Development and 

Validation of a Self-Assessment Instrument. In D. Hennessy, Traffic Psychology: An 

International Perspective (pp. 285–296). Nova Science Publishers, Inc.
20 

Appendix A: Railroad 1 Conditions for Learning Questions and 

Responses

This appendix shows the list of questions for Railroad 1’s Conditions for Learning Assessment. 

Questions were given with different scales. The below figures (Figure A1, Figure A2, and Figure 

A3) show the average response for each question as well as the minimum and maximum score 

each question could be rated by the students (0–3, 0–4, and 0–5). The figures differ by the 

maximum score.

Figure A1. Questions and Responses for Conditions for Learning Assessment: Railroad 1 

(Scale 0 to 3)

Figure A2. Questions and Responses for Conditions for Learning Assessment: Railroad 1 

(Scale 0 to 4) 

Figure A3. Questions and Responses for Conditions for Learning Assessment: Railroad 1 

(Scale 0 to 5) 
21 

Appendix B: Railroad 2 Conditions for Learning Questions and 

Responses

This appendix shows the list of questions for Railroad 2’s Conditions for Learning Assessment. 

The below figures (Figure B1, Figure B2, and Figure B3) show the average response for each 

question as well as the minimum and maximum score each question could be rated by the 

students (0–3, 0–4, and 0–5). The figures differ by the maximum score.

Figure B1. Questions and Responses for Conditions for Learning Assessment: Railroad 2 

(Scale 0 to 3)

Figure B2. Questions and Responses for Conditions for Learning Assessment: Railroad 2 

(Scale 0 to 4) 

Figure B3. Questions and Responses for Conditions for Learning Assessment: Railroad 2 

(Scale 0 to 5) 
22 

Abbreviations and Acronyms

ACRONYMS DEFINITION

ACCEL Accelerated Curriculum to Create Effective Learning

ACT Anticipate, Control, and Terminate

CFR Code of Federal Regulations

COL Conditions for Learning

C3RS Confidential Close Call Reporting System

FRA Federal Railroad Administration

FOCAL Forward Concentration and Attention Learning

LAG Less Aggressive Goals

NJT New Jersey Transit

PRT Peer Review Team

RAPT Risk Awareness and Perception Training

SBT Scenario-Based Training

STRAP Secondary Task Regulatory & Anticipatory Program

Volpe Center Volpe Transportation Systems Center