**SafeStat Effectiveness Study Update**

## 

## 

## March 2004

  ------------------------------- ---------------------------------------
  **Prepared for:**               **Prepared by:**

  **Federal Motor Carrier Safety  **John A. Volpe National Transportation
  Administration**                Systems Center**

  **400 Seventh Street SW**       **Motor Carrier Safety Assessment
                                  Division, DTS-47**

  **Washington, D.C. 20590**      **Kendall Square**

                                  **Cambridge, MA 02142**
  ------------------------------- ---------------------------------------

##  

## Background

In 1998, an effectiveness study was devised to confirm that motor
carriers that SafeStat identifies are indeed high safety risk carriers.
Safety risk at any given time is defined as the likelihood of having
crashes in the near future. By examining the SafeStat
post-identification crash experience of identified carriers, this study
essentially tested SafeStat's crash rate prediction capability and
represents the "bottom-line" assessment of its performance. Beyond
confirming SafeStat's effectiveness, the results of this study were used
to refine SafeStat to further emphasize the components of the system
that are the most closely related to high future crash rates. Since
1998, there have been changes made to the SafeStat algorithm, and in the
quality of data used by SafeStat, as well as changes to FMCSA motor
carrier safety program policy and in the motor carrier industry. To
better determine the current effectiveness of SafeStat in identifying
the high safety risk carriers, the effectiveness study has been updated
using the current version of the SafeStat algorithm (version 8.5) and
recent motor carrier data (available as of March 2003). The report
summarizes the results of this updated study.

## Description of the Effectiveness Study

The effectiveness study was accomplished by: (1) performing a simulated
SafeStat carrier identification using historical data; (2) observing the
crash involvement over the immediate 18 months after SafeStat was run
for both the carriers identified by SafeStat as having poor safety
status and other carriers not so identified by SafeStat, but which had
sufficient data to be identified; and (3) comparing the
post-identification crash rates of both groups of carriers. If SafeStat
is effective in identifying unsafe carriers (i.e., carriers having a
high risk of being involved in future crashes), then the carriers
identified as having a poor safety status would be expected to have
higher post-selection crash rates than the carriers that were not
identified by SafeStat. The greater the post-selection crash rate for
the identified carriers relative to those carriers not identified, the
more effective SafeStat would be in identifying unsafe motor carriers.

Rather than use the most recent SafeStat results and having to wait for
a period of time to collect post-identification crash data, the analysis
was performed using historical data. The study was conducted by
simulating a carrier identification by SafeStat on an earlier date
(March 24, 2001) and then observing the carriers' crash involvement that
occurred over the next 18 months (from March 2001 to September 2002).
This procedure simulated carrier identification by the current version
of SafeStat as if it had been run as of March 24, 2001 using safety
events that occurred prior to that date, and allowed enough time for
sufficient subsequent crash reporting to accurately measure the
post-identification crash rates. The study was performed using data
available as of March 2003. A relatively small number of carriers (150
out of 118,907) was excluded from the study due to erroneous data.

![](media/image1.wmf)

**Figure 1. Effectiveness Analysis Timeline**

From this simulation run of SafeStat, carriers that had sufficient data
to be scored were placed into the following groups based on their
overall SafeStat results in order to compare the "post-selection crash
performance":

1.  Carriers identified as "at-risk" (Category A and B Carriers),

2.  Other carriers identified as having a poor safety status according
    to SafeStat (Category C carriers), and

3.  Carriers with sufficient data but not identified by SafeStat as
    having a poor safety status

The post-identification crash rate of each group was calculated as the
number of reported crashes per 1000 power units (PUs). The number of PUs
is defined as the total number of trucks, tractors, hazardous-material
tank trucks, motor coaches, and school buses that are owned or leased by
a motor carrier. The carrier PU information was based on census data
that reside in the centralized FMCSA national database, the Motor
Carrier Management Information System (MCMIS).

The crash data were based upon the crashes reported by the states
(according to the National Governors' Association (NGA) standard) that
occurred during the post-selection period (March 2001 to September
2002). These data also reside in the MCMIS. Each reported crash was
weighted based on the severity and timing of the crash.

The severity weighting scheme placed emphasis on crashes with greater
consequences, while the time weighting placed emphasis on crashes that
occurred soon after the SafeStat identification run. Severity weights
were assigned as follows: a weight of 0.5 for property damage only, a
weight of 1.0 for crashes involving injuries/fatalities or hazardous
material release, and a weight of 1.5 for crashes involving
injuries/fatalities and hazardous material release. Time weights were
assigned to each crash as follows: a weight of 1.5 for crashes that
occurred within the first six months of 18 month post-selection time
period, a weight of 1.0 for crashes that occurred 7 to 12 months into
the post-identification time period, and a weight of 0.5 for crashes
that occurred in the last 6 months of the time period. Each crash had
its severity weight multiplied by its time weight to obtain an overall
weight. In each carrier group, the numbers of weighted crashes were
summed and divided by the number of total PUs to provide a weighted
crash rate for the group. The following section discusses the results
for each carrier group.

## Results

[Overall Effectiveness of SafeStat]{.underline}

The post-selection crash rates for the SafeStat identified and not
identified carrier groups were examined both in terms of their overall
SafeStat Scores and in terms of the four Safety Evaluation Areas (SEAs)
Accident, Driver, Vehicle, and Safety Management that determine the
overall SafeStat Scores. The rates are shown in Table 1.

**Table 1. Post-Selection Crash Rates**

  ---------------------------- ------------- ------------ ------------------
  **Carrier Group**            **Number of   **Weighted   **% Higher than
                               Carriers**    Crash        Not Identified
                                             Rate\***     Carriers**

  **All Identified**           **5,952**     **42.5**     **73%**

  At-Risk (Category A&B)       3,595         52.0         112%

  Other Identified (Category   2,357         29.4         20%
  C)                                                      

  **Not Identified**           **112,805**   **24.6**     **-**
  ---------------------------- ------------- ------------ ------------------

\* Number of weighted crashes per 1,000 power units from 3/2001 to
9/2002.

These results confirm that SafeStat did identify carriers with a higher
crash risk. The group of all carriers that SafeStat identified as poor
performers had a 73% higher crash rate than carriers that were not
identified. The carriers designated as "at-risk" by SafeStat had a much
higher crash rate (112% greater) than the carriers that were not
identified. A majority of these "at-risk" carriers were identified in
part because they had previous problems with respect to their crash
rates (i.e., they had deficient Accident SEA values).

The SafeStat-identified carriers in the "other identified" group
(Category C carriers), which did not have high Accident SEA values but
were in the worst 25th percentile in two of the other SEAs, posed a 20%
greater crash risk than the carriers that were not identified. This
result shows that SafeStat has the proactive capability to identify
carriers that are likely to be involved in crashes even though they
previously did not have exceptionally high crash rates.

[Effectiveness of Individual SEAs]{.underline}

Further testing was done to determine the effectiveness of the principal
components of SafeStat. This was accomplished by placing carriers into
groups based on their performance results for each particular SEA (i.e.,
Accident, Driver, Vehicle, or Safety Management (SM)).

The results for carriers with high individual SEA values compared to
those with lower SEA values are as follows (Carriers with [high SEA
values]{.underline} were in the worst 25th percentile and were
designated as the worst performers in that particular evaluation area.
Conversely, carriers with [no high SEA values]{.underline} were not in
the worst 25th percentile, and therefore, were not among the poorest
performers in that SEA.):

**Table 2. Crash Rates of Carriers with and without High SEAs**

+-------------------+-------------+-------------+----------------------+
| **Safety          | **Number of | *           | **% Greater than     |
| Evaluation Area** | Carriers**  | *Weighted** | Carriers without the |
|                   |             |             | High SEA**           |
|                   |             | **Crash     |                      |
|                   |             | Rate\***    |                      |
+-------------------+-------------+-------------+----------------------+
| High Accident SEA | 3,838       | 63.5        | 169%                 |
+-------------------+-------------+-------------+----------------------+
| No High Accident  | 114,919     | 23.6        | \-                   |
| SEA               |             |             |                      |
+-------------------+-------------+-------------+----------------------+
|                   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| High Driver SEA   | 12,391      | 39.2        | 63%                  |
+-------------------+-------------+-------------+----------------------+
| No High Driver    | 106,366     | 24.1        | \-                   |
| SEA               |             |             |                      |
+-------------------+-------------+-------------+----------------------+
|                   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| High Vehicle SEA  | 18,745      | 27.0        | 6%                   |
+-------------------+-------------+-------------+----------------------+
| No High Vehicle   | 100,012     | 25.4        | \-                   |
| SEA               |             |             |                      |
+-------------------+-------------+-------------+----------------------+
|                   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| High SM SEA       | 4,448       | 38.3        | 53%                  |
+-------------------+-------------+-------------+----------------------+
| No High SM SEA    | 114,309     | 25.0        | \-                   |
+-------------------+-------------+-------------+----------------------+

\* Number of weighted crashes per 1,000 power units from 3/2001 to
9/2002.

*Accident SEA* - The results confirm what may seem intuitively to be
obvious: carriers with high crash rates in the past are likely to
continue to have high crash rates in the future. In other words, past
crash rate performance is a good indicator of future crash rate
performance. The effectiveness study shows a 169% greater post-selection
crash rate for carriers with poor Accident SEAs compared to carriers
that were not identified as having poor Accident SEAs. Comparing SEAs,
the Accident SEA is by far the most effective SEA for identifying
high-risk carriers.

*Driver SEA* - The Driver SEA (with a 63% higher crash rate for carriers
with poor Driver SEAs) is the next most effective SEA. These results
from the study are especially impressive because the criteria for the
Driver SEA are based on violations and are independent of crash history.

*Vehicle SEA* - Carriers with poor Vehicle SEAs did have a slightly
higher crash rate (6%) than carriers without poor Vehicle SEAs. This
result is much lower than the Driver SEA, thus implying that there is a
stronger relationship between driver violations and crash risk than
vehicle violations.

*Safety Management SEA* - The Safety Management SEA is also effective in
identifying carriers with high crash rates. Indicators in this SEA are
based on safety regulation compliance supporting the association of
safety regulations with crash risk. Carriers with high Safety Management
SEAs had a 53% higher post-identification crash rate than carriers that
did not have high Safety Management SEA values.

[Effectiveness of Individual Indicators]{.underline}

Further testing was done to determine the effectiveness of indicators of
SafeStat. The indicators are used to derive the SEA values. The follow
table shows the results for indicators for the carriers in the worst
25^th^ percentile.

**Table 3. Crash Rates of Carriers with High Indicators (\>=75)**

+-------------------+-------------+-------------+----------------------+
| **Indicator \>=   | **Number of | *           | **% Greater than     |
| 75**              | Carriers**  | *Weighted** | Carrier Population** |
|                   |             |             |                      |
|                   |             | **Crash     |                      |
|                   |             | Rate\***    |                      |
+-------------------+-------------+-------------+----------------------+
| Accident          | 3,811       | 70.0        | 174%                 |
| Involvement       |             |             |                      |
| Indicator (AII)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Recordable        | 489         | 35.5        | 39%                  |
| Accident          |             |             |                      |
| Indicator (RAI)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Driver Inspection | 10,727      | 37.9        | 48%                  |
| Indicator (DII)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Driver Review     | 1,573       | 46.4        | 82%                  |
| Indicator (DRI)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Moving Violation  | 5,468       | 63.3        | 148%                 |
| Indicator (MVI)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Vehicle           | 18,736      | 27.0        | 6%                   |
| Inspection        |             |             |                      |
| Indicator (VII)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Vehicle Review    | 23          | 27.6        | 8%                   |
| Indicator (VRI)   |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Enforcement       | 1,789       | 39.5        | 54%                  |
| History Indicator |             |             |                      |
| (EHI)             |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Safety Mgmt.      | 3,016       | 35.4        | 39%                  |
| Review Indicator  |             |             |                      |
| (SMRI)            |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| Hazardous         | 374         | 35.0        | 37%                  |
| Material Review   |             |             |                      |
| Indicator (HMRI)  |             |             |                      |
+-------------------+-------------+-------------+----------------------+
| **Carrier         | **118,757** | **25.6**    | **-**                |
| Population**      |             |             |                      |
+-------------------+-------------+-------------+----------------------+

\* Number of weighted crashes per 1,000 power units from 3/2001 to
9/2002.

The Accident Involvement Indicator (AII), based on state-reported
crashes, is clearly the indicator with the strongest link to future
crash risk. The newest indicator, the Moving Violation Indicator (MVI),
based on moving violations recorded during roadside inspections, is very
close to the AII in it effectiveness with identifying high risk
carriers. The Vehicle Inspection Indicator (VII) had nominal value in
identifying high risk carriers, but it identified the largest portion of
carriers (18,736).

## 

## Comparison with the 1998 Effectiveness Study

When comparing results of the original effectiveness study to the new
results, it is clear that the original results showed that SafeStat was
identifying carriers with relatively higher crash rates. The original
study showed the At-Risk Group and Other Identified Group with future
crash rates that were 169% and 41% higher, respectively, than the group
of carriers not identified (compared to 112% and 20% for the same groups
in the new study -- see Table 1). There is, however, a major change in
the environment from the original study conducted 1998 to this new
study. The pool of carriers used in the original study was operating in
a "pre-SafeStat" environment. During the timeframe of the new study, the
carriers identified by SafeStat as potential high crash risk carriers
were subject to targeting by federal and state safety programs, such as
compliance reviews, road inspections, the PRISM program, and SafeStat
On-line. The main purpose of these programs is to prevent future crashes
from occurring. The effectiveness of these SafeStat-influenced safety
programs in reducing crashes does to some extent mute the effectiveness
of SafeStat to identify high crash risk carriers as measured in this
study. The effectiveness of the SafeStat-influenced programs is
significant. According to the a longitudinal study[^1] conducted over a
similar timeframe as the new effectiveness study, Category A and B
carriers reduced their crash rate by 45% over a period of 18 months
after they were identified in SafeStat. Repercussions of these programs
on SafeStat-targeted carriers can lead to companies leaving the
industry. The results of the new study showed that 21% of at-risk
carriers (and 14% of the other identified carriers) were no longer
actively operating interstate commercial motor vehicles after the 18
months from the March 2001 SafeStat identification. These numbers are
significantly higher than the attrition rate of 5% of the carriers not
identified by SafeStat.

## 

## Conclusion

Despite these complications, the new effectiveness study is still
showing that SafeStat does work. The individual parts of SafeStat and
SafeStat as a whole do identify carriers that are likely to have
significantly higher crash rates than carriers not identified.

[^1]: Volpe National Transportation Systems Center, *Measuring the
    FMCSA's Safety Objectives from Year 2000 to 2002*, July 2003, pg
    6-3.
    (<http://ai.volpe.dot.gov/CarrierResearchResults/CarrierResearchResults.asp?file=PDFs/StrategicPlan.pdf>)