RR 23-18 | September 2023

SHORT LINE SAFETY INSTITUTE: TESTING THE 

FIDELITY OF ITS ROBUST MODEL

SUMMARY

The Short Line Safety Institute (SLSI) is 

dedicated to the continuous improvement of 

safety and safety culture across all short line 

and regional railroads in the United States. 

Strengthening a railroad’s safety culture may 

result in less frequent or severe accidents and 

create a safer working environment. SLSI 

defines safety culture as the shared values, 

actions, and behaviors that demonstrate a 

commitment to safety over competing goals and 

demands, based on the U.S. Department of 

Transportation (DOT) Safety Council’s safety 

culture definition (Morrow & Coplen, 2017).

SLSI conducted a study to assess the fidelity of 

its Safety Culture Assessment (SCA) model. 

This report summarizes findings from this study

and discusses areas for enhancement of the 

assessment process. 

BACKGROUND

Since 2015, SLSI has conducted voluntary, nonpunitive, and confidential assessments of the 

safety culture at participating short line and 

regional freight railroads (i.e., Class II and Class 

III railroads) across the United States. SLSI uses

a multi-method model that has been recognized

as “the most robust assessment model in the 

industry” by a Volpe National Transportation 

Systems Center (Volpe) evaluation (Kidda & 

Howarth, 2019). SLSI’s SCA model, which 

includes interviews, on-site observation, and 

surveys, continues to provide tangible, actionoriented results for participating railroads.

SLSI uses independent contractor Assessors to 

conduct the SCAs. These Assessors are subject 

matter experts and highly dedicated to their work 

with SLSI.

OBJECTIVE

The objective of the fidelity assessment is to

assess how well SLSI’s Assessors adhere to the 

protocols of the SCA model to evaluate a 

railroad's safety culture. It also identifies any

room for improvement in the SCA process, 

protocol language, and future Assessor training. 

METHODS

SLSI observed each process of the SCA model 

for various locations and Assessor teams. This 

included large (>25 employees) and small (<25

employees) railroads and activities conducted in 

person and over the phone. A total of seven

different locations and seven Assessors were 

observed. SLSI evaluated whether each 

Assessor adhered to the protocols established 

for each part of the SCA from Readiness 

Screening to Post-Assessment Call. Measures 

included “yes/no” agreement observations as 

well as “very high/very low” conformance scales. 

Interviewees and Assessors were also given a 

questionnaire to measure their engagement with

and sentiments toward the overall SCA process. 

SCA Elements. The SCA includes various 

elements such as Advanced Coordination 

Meetings, Kick-Off Meetings, Interviews, 

Observations, Document Inventories, Close-Out 

Meetings, Post-Assessment Calls, and 

Feedback Surveys. Each element of the SCA 

includes a step by step written protocol that is 

available to all Assessors. Additional tools are 

also provided as needed (e.g., report writing 

templates and observation checklists). 
RR 23-18 | September 2023 

RESEARCH RESULTS 2 | Page

ANALYSIS

Data analysis included content coding of

interviewee and Assessor engagement 

interviews. Frequencies of emergent themes 

were then calculated as percentages. Volpe 

conducted an analysis of the agreement and 

conformance scores produced for each SCA 

element observed. Lastly, SCSI carefully 

reviewed all protocols and documents. 

RESULTS

Frequency analyses revealed the prevalence of 

protocol adherence and deviation. A thorough 

evaluation of all protocols and documents also

highlighted areas for improvement. Key results 

are listed below.

Figure 1 and Figure 2 show the aggregated 

results for each of the SCA activities measured 

across three railroads. While the fidelity was 

measured for seven railroads, only three of the 

railroads contained data for all assessment 

activities. The remaining four railroads were 

evaluated only for the pre-assessment and postassessment activities. Since the scores were all 

100 percent for the completion scores and not 

completed for the agreement scores, their data 

would not change either figure’s data.

Figure 1. Compliance scores for each SCA 

element

Figure 1 shows the percent completion scores 

for activities involving a binary task (i.e., 

completed or uncompleted). Of the 10

assessment activities, only 2 showed less than 

100 percent completion. Interviews showed a 

95.8 percent completion score while the CloseOut Meeting showed a completion score of 96.3

percent. These were the only activities for which 

the completion score was less than 100 percent. 

Figure 2. Agreement scores for SCA elements

Figure 2 shows agreement scores for three of 

the activities where assessors may diverge in 

their judgements. These activities included 

interviews, field observations, and analysis of 

the assessment data. All three activities 

registered scores over 4 on a scale where 0 

represents no agreement among Assessors and 

5 represents complete agreement. The data 

suggests that the Assessors mostly agreed in 

their judgments about each railroad’s interviews, 

observations, and data analysis.

Overall, this data suggests that the Assessors 

closely follow the procedures designed to 

assess a railroad’s safety culture. Opportunities 

for improving the process were found in the 

Close-Out Meeting and Interview activities. 

Fidelity Strengths

• Protocol Compliance and Agreement: SLSI

Assessors mostly complied with all SCA

protocols and procedures.

• Assessor Engagement: SLSI Assessors (N

= 7) expressed confidence in the SCA

process and protocols (100 percent). They

considered their interactions with railroads

valuable to both themselves (100 percent)

and the industry (100 percent). All

Assessors provided ideas for improving the

SCA process (e.g., additional Assessor

training and networking events).
RR 23-18 | September 2023 

RESEARCH RESULTS 3 | Page

• Interviewee Engagement: Interviewees (N =

31) expressed satisfaction with the interview

process (100 percent), confidence in SLSI’s

dedication to confidentiality (100 percent),

and were comfortable with SLSI being on

their property (83.87 percent). Only 16.13

percent of interviewees provided ideas for

improving the interview process. These

ideas mostly concerned the length of some

of the interview questions.

Fidelity Opportunities for Improvement

• Updating of Protocols: Some protocols and

documents were out of date, listing

procedures that were no longer in place and

information that was no longer accurate.

Updating these protocols will assist in

providing all Assessors with the most up-todate information and maintaining

consistency among Assessor behaviors.

• Conformity of Protocols: Some protocols

and documents included differing formats,

fonts, headers, and footers. Conformity of

these protocols will assist in the overall

visual professionalism of the SCA process

and could help Assessors better organize

their documents.

• Creating New Protocols: The Observation

element of the SCA model includes vehicle,

facility, and work observations. There is

currently one protocol listed, but a specific

protocol for each of the three observations

may provide additional clarity for each

activity.

• Assessor Refresher Training: SLSI could

provide training to remind Assessors of

protocols, locations of documents, and the

use of documents, also receiving feedback

from the Assessors. During future training,

Assessors could be refreshed on Close-Out

Meeting and Interview protocols to ensure

that these protocols are being carefully

followed.

CONCLUSIONS

SLSI’s fidelity assessment suggests there is a 

high Assessor conformance to SCA protocols 

and procedures. However, there are also some

areas for improvement. By assessing its fidelity

and examining the practices of its Assessors, 

SLSI can gain further insight into its robust 

process. With this information, SLSI can fine 

tune its protocols, procedures, and Assessor 

training, fulfilling its goal of continuous 

improvement. 

FUTURE ACTION

This fidelity assessment highlights opportunities 

for improvement for SLSI. SLSI continues to 

update and manage its protocols to provide the 

most accurate information to the Assessors and 

maintain consistency in Assessor behaviors 

across SCAs. 

SLSI also continues to provide annual Assessor 

training. Findings from this study can be shared 

with Assessors and used as a tool during this 

training. 

REFERENCES

Kidda, S., & Coplen, M. (2016). Short Line 

Safety Institute Pilot Safety Culture

Assessment Project: Development of

Assessment Tools (Report No. RR 16-03). 

Federal Railroad Administration.

Kidda, S., & Howarth, H. D. (2019). Short Line 

Safety Institute: The Most Robust Model for 

Assessing Safety Culture in the US Railroad 

Industry (Report No. RR 19-15). Federal 

Railroad Administration. 

Morrow, S., & Coplen, M. (2017). Safety Culture: 

a Significant Influence on Safety in 

Transportation (Report No. DOT/FRA/ORD17/09). Federal Railroad Administration
RR 23-18 | September 2023 

RESEARCH RESULTS 4 | Page

CONTACT

Starr Kidda, Ph.D.

Human Factors Division Chief 

Federal Railroad Administration

Office of Research, Development, and

Technology

1200 New Jersey Avenue, SE

Washington, DC 20590

(202) 306-2011

Starr.Kidda@dot.gov

Julia Leone, Ph.D.

Manager of Research and Organizational 

Development

Short Line Safety Institute

50 F Street, N.W., Suite 500 

Washington, DC 20001

(202) 628-5349

Julia.Leone@shortlinesafety.org

KEYWORDS

freight railroads, regional railroads, short line 

railroads, safety culture assessment, safety 

culture measurement, state of rail industry

CONTRACT NUMBER

F33315-86-C-5169