USA Banner
  U.S. flag
              An official website of the United States government Here's how you knowOfficial US Government Icon
              **Official websites use .gov**A **.gov** website belongs to an official government organization in the United States.
            Secure Site Icon
              **Secure .gov websites use HTTPS**
                  A **lock** ( A locked padlock ) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. 
 U.S. Department of Transportation Icon United States Department of Transportation
 Search DOT
            Search
                    About FHWAFHWA Organization
                Headquarters Offices
                Field Offices
                Strategic Plan
                Business Opportunities
                Staff Directories
                Careers
                Highway History
                Contact UsProgramsAcquisition Management
                Civil Rights
                Federal Lands Highway
                Infrastructure
                Innovative Program Delivery
                Operations
                Planning, Environment, and Realty
                Policy
                Research, Development, and Technology
                Safety
                Browse by TopicsResourcesCore Highway Topics
                Federal Aid Essentials
                Automation
                Complete Streets
                Laws and Regulations
                Policy & Guidance Center
                Section 508
                Publications and Statistics
                Professional Development
                National Highway Institute
                Resource Center
                Accessibility Resource Library
                FHWA Research LibraryNewsroomPress Releases
                Speeches & Testimony
                Photos
                Videos
                Media ContactsMenu FHWA Highway Safety Programs Home
                        Improving Safety on Rural Local and Tribal Roads Safety Toolkit


            
            In This Section
        Local and Rural Road Safety Program
                Training, Tools, Guidance and Countermeasures for Locals Practitioners
                Crash Facts
                
                Step 7. Evaluate Effectiveness
Step 7. Evaluate Effectiveness

Overview

The purpose of this step is to describe how to evaluate the impact of the treatments that have been implemented in terms of crash frequency or severity. A reliable assessment of the effectiveness of safety countermeasures cannot be made immediately after implementation. Some time needs to pass, often two to three years, before enough data can be collected to determine how many crashes, serious injuries, and fatalities have occurred since implementation of the countermeasure and then compare it with the same types of data from before implementation.

This step should not be overlooked. Evaluation provides information that can help agencies decide whether or not the investment has reduced crash frequency or severity. Evaluation also can help agency staff demonstrate the value of the program to community leaders and the general public.

The level of effort required to conduct an evaluation depends on the resources available and the method chosen. Some methods are:

Collect public feedback data; such as:
	The number of complaints received about the location of interest;
		The number of compliments received on the installed countermeasure(s);
		The number and type of police citations issued; and
		The number and type of maintenance issues at the site.
	Conduct a comparative assessment of before and after crash frequency, severity, and traffic volumes.
	Conduct a simple before/after study.
	Conduct a rigorous before/after analysis.At the most basic level, evaluation is a “before and after” comparison. This means that some sort of “before” or “baseline” data must be available against which the “after” data can be compared. This baseline data is often available as part of the information used to select the site in the first place. To demonstrate the benefits of safety investments, the agency should track before and after performance measures, analyze trends, and conduct selective benefit/cost analysis. The Highway Safety Improvement Program manual provides detailed information about how to do this. Please see the Resources at the end of this section.

Example

**Comparative Assessment – Treatment at One Site.** Figure 17 illustrates a comparative analysis of before and after conditions conducted after a treatment has been implemented at a single site.

Charts A, B, and C summarize crash counts, average daily traffic, and target crash type crash counts for the three years before and after implementing the project. Charts D and E show the relative (percent) and absolute (number) changes in traffic volumes and crashes after implementation. Visually comparing these characteristics side by side allows a qualitative assessment of the treatment’s effectiveness. A comparative assessment considers crash counts, crash type and severity and traffic volume to develop a qualitative assessment of the changes from before to after deploying a treatment.

Figure 17. Example of Comparative Analysis of Before and After Conditions

Figure 17 shows a graphic representation of crashes by type and year, average daily traffic volume, target crash frequency by year, percent and count in change in traffic volume, target crashes, and crash severity. Collectively these graphs can be used to evaluate the change in site conditions after implementing a treatment.

*Source: Oregon Department of Transportation, Evaluating the Effectiveness of the Safety Investment Program (SIP) Policies for Oregon.*

**Simple Before/After Crash Analysis.** A simple before/after crash analysis compares the number of crashes that occurred during the three-year period before a treatment is installed to the number of crashes that occurred in the three-year period after the treatment is installed (see Table 11). A simple before/after crash analysis is focused only on crash counts for the purposes of estimating the quantitative benefits of deploying a treatment.

Table 11. Example Simple Before/After Analysis
	2006-2008
			2009-2011
			Percent ReductionTotal Crashes at Site
			22
			16
			27%This type of analysis is an appealing way to estimate the quantitative safety benefits of a treatment because it is easy to complete, requires little data, and is easy to explain. But there are some important limitations to it. The main problem with it stems from the underlying assumption that the number of crashes experienced before implementing the countermeasure is a good estimate of the number of crashes that would be expected in the future assuming the countermeasure had not been implemented. However, this is not always the case because traffic volumes or surrounding land uses may change or levels of enforcement may change. Due to this and other limitations, it cannot be relied upon solely to evaluate effectiveness.

**Rigorous Before/After Crash Analysis.** To address the weaknesses inherent in simple before/after crash analysis, methods that utilize more sophisticated statistical tools are used. These include before/after analysis with comparison groups, empirical Bayes (EB) analysis, full Bayes studies, and cross-sectional studies. These methods should be used whenever an agency wants to develop a reliable quantitative estimate the effectiveness of a treatment (to develop a Crash Modification Factor (CMF) for example). The Highway Safety Manual and “Recommended Protocols for Developing Crash Modification Factors” available on the Crash Modification Factor Clearinghouse web site provide information and guidance on using these methodologies. Please see the Resources at the end of this section.

**Empirical Bayes Analysis – What is it?**

The Empirical Bayes (EB) method is a statistical method that combines the observed crash frequency with the predicted crash frequency using SPFs (see below) to calculate expected crash frequency. By combining crash history and predicted crash frequency, a better estimate of expected crash frequency can be made.

**SPF (safety performance function)** represents the change in mean crash frequency as ADT (or other exposure measure) increases or decreases.

The EB method overcomes regression-to-the-mean.

Application

Effectiveness evaluations are typically conducted two to three years after the treatment has been in place. There are three primary reasons for this:

It takes time for enough data to accumulate so that meaningful results can be obtained.
	Road users, who may be quite familiar with the roadway environment “pre-treatment,” will often modify their driving behavior “post-treatment” until they become familiar with the new environment. Because this interim period does not reflect the long-term impact of the treatment, it is important to wait until drivers are used to the new roadway environment before evaluating effectiveness.
	In contrast, if the evaluation is conducted too many years after the treatment is implemented, conditions such as traffic volumes and land uses around the site may have changed to such a degree that any observed change in crash frequency or crash severity may not be attributable *only* to the treatment.Resources For Step 7: Evaluating Effectiveness

Highway Safety Improvements Program Manual

Chapter 6 of the Highway Safety Improvement Program (HSIP) manual discusses various safety evaluation methods and provides calculation example for each method. The manual’s Table 6.1 compares data and inputs required for these methods.

The report is FHWA Report Number: FHWA-SA-09-029.

A Guide to Developing Quality Crash Modification Factors

Cover of Federal Highway Administration document: A Guide to Developing Quality Crash Modification Factors.

*Source: FHWA.*

This FHWA guide provides guidance for users to develop crash modification factors (CMF). Table 12 in the guide provides users with a comparison of evaluation study designs. The guide’s Chapter 4 provides a flowchart for users to select preferred evaluation design based on data availability and project.

The report is FHWA Report: https://www.cmfclearinghouse.org/collateral/CMF_Guide.pdf.

Highway Safety Manual

Volume 1, Chapter 9 of the HSM documents and discusses the various methods for evaluating the effectiveness of a treatment, a set of treatments, an individual project, or a group of similar projects after safety improvements have been implemented. The chapter also highlights which methods are appropriate for assessing safety effectiveness in specific situations, and provides step-by-step procedures for conducting safety effectiveness evaluations.

National Highway Institute (NHI) Highway Safety Improvement Program (HSIP) Project Evaluation

This is a NHI course that presents users with a description of safety effectiveness evaluation, an overview of fundamentals for performing safety effectiveness evaluation, and information about the importance of safety effectiveness evaluation in the context of the HSIP. Users are provided example before and after studies, and learn about the data needs for each methodology. The course lasts five hours and is partially instructor led and partially web based.

More course information is available on the National Highway Institute’s web site.

You may need Adobe® Reader® to view the PDFs on this page.
      
  Book traversal links for Step 7. Evaluate Effectiveness
      ‹ Step 6. Implement Countermeasures
                    Up
                    Conclusion ›
                
                Book:Improving Safety on Rural Local and Tribal Roads Safety Toolkit
              Introduction
                How to Use the Toolkit and User Guides
                Step 1. Compile Data and Resources
                Step 2. Conduct Network Screening
                Step 3. Select Sites for Investigation
                Step 4. Diagnose Site Crash Conditions and Identify Countermeasures
                Step 5. Prioritize Countermeasures for Implementation
                Step 6. Implement Countermeasures
                Step 7. Evaluate Effectiveness
                Conclusion U.S. DEPARTMENT OF TRANSPORTATION

**Federal Highway Administration**

1200 NEW JERSEY AVENUE, SE

WASHINGTON, DC 20590

202-366-4000Subscribe to email updates
                    About
  

        
              About FHWA
                Careers
                Org Chart
                Staff Directories
                Work with UsPrograms
  

        
              Acquisition Management
                Civil Rights
                Federal Lands Highway
                Infrastructure
                Innovative Program Delivery
                Operations
                Planning, Environment, and Realty
                Policy
                Research, Development, and Technology
                SafetyPolicies, Rights, Legal
  

        
              About DOT
                Accessibility
                Budget and Performance
                Civil Rights
                FOIA
                Information Quality
                No FEAR Act
                Office of Inspector General
                Privacy Policy
                USA.gov
                Web Policies and Notices
                Web StandardsNews & Events
  

        
              Newsroom
                Press Releases
                Speeches & Testimony
                Media Contacts
                Connect with Us