# Presentation: 287377

## Risk Reduction Program

- Michael Coplen
- Senior Evaluator
- Office of Railroad Development
- Human Factors R&D Program
*Safety Summit Meeting*
*12 August 2008*

## Data driven evaluation to ensure RRP success

- Who are the primary stakeholders, what are their information needs, and how can we support key policy and decision making?
- How can we ensure effective implementation of RRP pilots?
- How do we determine program impact and effectiveness?
- How can we increase the likelihood that program products and processes will be used?
- We answer all these questions with *program evaluation.*

## What is Program Evaluation?

- Program evaluation systematically collects information about a program’s stakeholders, implementation, outcomes & impacts.
- Knowledge about how to do **useful program evaluation **comes from a field of professional practice that has developed over the last 35 years and includes:
- The American Evaluation Association & International Organization for Cooperation in Evaluation
- Annual professional conferences, scholarly journals, standards, guidelines, training courses, & expertise
- Evaluation includes multiple methods to systematically assess program performance & effectiveness, paying special attention to distinguishing **formative evaluation **(learning for continuous improvement) from **summative evaluation **(overall judgments about merit & worth to inform decisions).

**Notes:**
 Involve cross-cutting methodologies and disciplines (quantitative and qualitative)
 Systematically measure program performance (activities, processes and outcomes)
 Documents accountability (ascertaining its strengths weaknesses and inherent ‘value’;
A formative process to guide decision-making and provide direction for continuous improvement;
A strategic and multi-disciplinary process that promotes appropriate and effective use of findings; 

## Key Elements of Program Evaluation

- Logic Models
- Show critical program elements and their relationships
- Constructed in collaboration with stakeholders
- Visually depicts implementation design, the program theory of change, and its intended outcomes
- Program Metrics
- Include both quantitative and qualitative data
- Helps determine implementation effectiveness
- Highlight early indicators of success
- Methodology
- Measure causal relationships among program elements, their intended and unintended outcomes, to determine program impacts
- Utilization and sustainment strategies
- Provide timely and effective communication of program results throughout entire lifecycle
- Build organizational commitment for short and long-term impacts

## RRP Implementation Logic Model

## Individual Impact of Peer-to-Peer Observation and Feedback

**Notes:**
When people talked about sampling, few talked about being sampled and having that change their behavior.  One talked about someone “catching” him not calling out his speed and now he does.  Another talked about doing quite a bit of self-sampling early on because he wanted to improve his own behavior.  But generally, the theme that surfaced was that of sampling others.  Whether in actual 3rd-party sampling role or crew sampling.  Or even in a informal sampling role.

The overt change described here is how this single observation has affected change in the Old Head.  This engineer could be quite a bit younger than the old head, but because he knows him well, and has knowledge of this person, he uses that knowledge to impact an old head.  (Program participants have the most credibility in implementing change in their environment)
We know that the actual process of Sampling involves a whole lot more steps than he describes, but this engineer  is clearly acting the role of a sampler.  Less overt is that he has internalized the process and is generalize it to his everyday job and is doing it on his own.  He owns the CAB process in his daily activity.

## Individual Impacts of CAB Class: Internalization & Generalization

- 	“I tell ya, the CAB class has made me start doing something that I never did before - I put safety glasses on when I weed eat or cut the grass.  And, it's strictly just the CAB class that's done it.  You know, and it doesn't have anything to do with the railroad but it's kinda the mindset that everybody's going to you know, I gotta think about what I'm doing before I do it.  Period.  It doesn't matter if it's at work or at home or you know driving down the road.” [Mgr 15 yrs]

**Notes:**
Almost half of the people told us that the lessons they are learning on the job are transferring to their lives off work, or off the yard. From hanging ceiling fans to standing on ladders, most everyone we spoke to provided stories of how they’re safer now than before CAB.  One person related a story about washing his truck in his sloped driveway.  He’d never thought of it before, but when he brought out the ladder to get up and wash the truck hood he thought twice, tested the ladder and found it unsafe.  So, he had to find another way to wash the hood.

This takes us to the 2nd order outcomes.  

## Impact of CAB Program

*“**We don’t circle the wagons around a fellow employee anymore just to protect them. If they are behaving unsafely, we want them to change their behavior or we want them outta here, because they are putting the rest of us at risk.” [Eng 14 yrs]*

**Notes:**
When people talked about sampling, few talked about being sampled and having that change their behavior.  One talked about someone “catching” him not calling out his speed and now he does.  Another talked about doing quite a bit of self-sampling early on because he wanted to improve his own behavior.  But generally, the theme that surfaced was that of sampling others.  Whether in actual 3rd-party sampling role or crew sampling.  Or even in a informal sampling role.

The overt change described here is how this single observation has affected change in the Old Head.  This engineer could be quite a bit younger than the old head, but because he knows him well, and has knowledge of this person, he uses that knowledge to impact an old head.  (Program participants have the most credibility in implementing change in their environment)
We know that the actual process of Sampling involves a whole lot more steps than he describes, but this engineer  is clearly acting the role of a sampler.  Less overt is that he has internalized the process and is generalize it to his everyday job and is doing it on his own.  He owns the CAB process in his daily activity.