Motor Carrier Safety: A Statistical Approach Will Better Identify Commercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach
Published: 2007-06-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA) has the primary federal responsibility for reducing crashes involving large trucks and buses that operate in interstate commerce. FMCSA decides which motor carriers to review for compliance with its safety regulations primarily by using an automated, data-driven analysis model called SafeStat. SafeStat uses data on crashes and other data to assign carriers priorities for compliance reviews. GAO assessed (1) the extent to which changes to the SafeStat model could improve its ability to identify carriers that pose high crash risks and (2) how the quality of the data used affects SafeStat's performance. To carry out its work, GAO analyzed how SafeStat identified high-risk carriers in 2004 and compared these results with crash data through 2005.
section_title: What GAO Found
paragraphs: While SafeStat does a better job of identifying motor carriers that pose high crash risks than does a random selection, regression models GAO applied do an even better job. SafeStat works about twice as well as (about 83 percent better than) selecting carriers randomly. SafeStat is built on a number of expert judgments rather than using statistical approaches, such as a regression model. For example, its designers decided to weight more recent motor carrier crashes twice as much as less recent ones on the premise that more recent crashes were stronger indicators of future crashes. GAO estimates that if FMCSA used a negative binomial regression model, FMCSA could increase its ability to identify high-risk carriers by about 9 percent over SafeStat. Carriers identified by the negative binomial regression model as posing a high crash risk experienced 9,500 more crashes than those identified by the SafeStat model over an 18 month follow-up period. The primary use of SafeStat is to identify and prioritize carriers for FMCSA and state compliance reviews. FMCSA measures the ability of SafeStat to perform this role by comparing the crash rate of carriers identified as posing a high crash risk with the crash rate of other carriers. Using a negative binomial regression model would further FMCSA's mission of reducing crashes through the more effective targeting of compliance reviews to the set of carriers that pose the greatest crash risk. Late-reported, incomplete, and inaccurate data reported to FMCSA by states have been a long-standing problem. However, GAO found that late reported data had a small effect on SafeStat's ability to identify carriers that pose high crash risks in 2004. If states had reported all crash data within 90 days after occurrence, as required by FMCSA, a net increase of 299 carriers (or 6 percent) would have been identified as posing high crash risks of the 4,989 that FMCSA identified. Reporting timeliness has improved, from 32 percent of crashes reported on time in fiscal year 2000, to 89 percent in fiscal year 2006. Regarding completeness, GAO found that data for about 21 percent of the crashes (about 39,000 of 184,000) exhibited problems that hampered linking crashes to motor carriers. Having complete information on crashes is important because SafeStat treats crashes as the most important factor for assessing motor carrier crash risk, and crash information is also the crucial factor in the statistical approaches that we employed. Regarding accuracy, a series of studies by the University of Michigan Transportation Research Institute covering 14 states found incorrect reporting of crash data is widespread. GAO was not able to quantify the effect of the incomplete or inaccurate data on SafeStat's ability to identify carriers that pose high crash risks because it would have required gathering crash records at the state level--an effort that was impractical for GAO. FMCSA has acted to improve crash data quality by completing a comprehensive plan for data quality improvement, implementing an approach to correct inaccurate data, and providing grants to states for improving data quality, among other things.

section_title: Letter
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA) within the U.S.  The Federal Motor Carrier Safety Administration (FMCSA) within the U.S.  Department of Transportation has the primary federal responsibility for  Department of Transportation has the primary federal responsibility for  reducing crashes, deaths, and injuries involving large trucks and buses  reducing crashes, deaths, and injuries involving large trucks and buses  operating in interstate commerce. While it carries out a number of  operating in interstate commerce. While it carries out a number of  activities toward this end, an important tool at its disposal is the  activities toward this end, an important tool at its disposal is the  compliance review—a detailed inspection of a motor carrier’s operations  compliance review—a detailed inspection of a motor carrier’s operations  at its place of business. FMCSA decides which carriers to inspect primarily  at its place of business. FMCSA decides which carriers to inspect primarily  by using an automated, data-driven analysis system called the Motor  by using an automated, data-driven analysis system called the Motor  Carrier Safety Status Measurement System (SafeStat). SafeStat uses data  Carrier Safety Status Measurement System (SafeStat). SafeStat uses data  on crashes, vehicle and driver violations, and other information to develop  on crashes, vehicle and driver violations, and other information to develop  numerical scores for carriers, and then SafeStat assigns each carrier a  numerical scores for carriers, and then SafeStat assigns each carrier a  priority to receive a compliance review.  priority to receive a compliance review.
Following an incident in which a bus company, with many driver  Following an incident in which a bus company, with many driver  violations and a low priority for compliance review from the SafeStat  violations and a low priority for compliance review from the SafeStat  model, suffered a fire on one of its buses that resulted in 23 deaths, you  model, suffered a fire on one of its buses that resulted in 23 deaths, you  were interested in whether SafeStat could better identify commercial  were interested in whether SafeStat could better identify commercial  motor carriers at risk for crashes. To address your interest, we assessed  motor carriers at risk for crashes. To address your interest, we assessed  (1) the extent to which changes to the SafeStat model could improve its  (1) the extent to which changes to the SafeStat model could improve its  ability to identify these carriers and (2) how the quality of the data used  ability to identify these carriers and (2) how the quality of the data used  affects SafeStat’s performance. These two topics are the main focus of this  affects SafeStat’s performance. These two topics are the main focus of this  report. We also examined the findings of other studies on how SafeStat’s  ability to identify carriers at risk for crashes can be improved. (See app. I.)
To determine whether statistical approaches could be used to improve  FMCSA’s ability to identify carriers that pose high crash risks, we tested a  number of regression models and compared their performance with  SafeStat’s results from June 2004. We chose 2004 because it allowed us to  examine actual crash data for the 18-month period following June 2004 to  determine the degree to which SafeStat successfully identified carriers  that proved to be of high risk for crashes. It also allowed us to include  crashes that occurred within the 18 months after June 2004 but had not yet  been reported to FMCSA by December 2005. Using regression models, we  compared the predictive performance of these statistical approaches to  SafeStat’s performance to determine which method best identified carriers  that pose high crash risks. We also calculated crash rates from a series of  random samples of all carriers to determine if the SafeStat model did a  better job than random selection in identifying motor carriers that pose  high crash risks. To assess whether changes could be made to the SafeStat  model to improve its identification of carriers that pose high crash risks,  we tested changes to selected portions of the SafeStat model and  investigated the effect of changing decision rules used to construct the  four safety evaluation areas.
To assess the extent to which data quality affects SafeStat’s ability to  identify carriers that pose high crash risks, we carried out a series of  analyses and surveyed the literature to identify findings from other  studies. To address timeliness, we measured the number of days it took  states to report crashes. We also added late-reported crashes to FMCSA’s  June 2004 data and recalculated SafeStat scores to determine the effect of  late-reported crashes on carriers’ rankings. For completeness, we  attempted to match crash records in FMCSA’s Motor Carrier Management  Information System (MCMIS) crash master file to motor carriers listed in  the MCMIS census file and reviewed studies on state reporting. To address  accuracy, we reviewed a report that tested the accuracy of electronic data  on a sample of paper records and studies that identified the impact of  incorrectly reported crashes in individual states on MCMIS data quality.  While there are known problems with the quality of the crash data  reported to FMCSA for use in SafeStat, we determined that the data were  of sufficient quality for our use, which was to compare the ability of  regression models to identify carriers that pose high crash risks to the  current approach, which is largely derived through professional judgment.  We conducted our work in accordance with generally accepted  government auditing standards from May 2006 through May 2007.  Appendix II provides further information on our scope and methodology.
Shortly, we expect to issue a related report that examines how FMCSA  identifies and takes action against carriers that are egregious safety  violators. In addition, that report examines how thoroughly and  consistently FMCSA conducts compliance reviews.
subsections: section_title: Background
paragraphs: The interstate commercial motor carrier industry, primarily the trucking  industry, is an important part of the nation’s economy. Trucks transport  over 11 billion tons of goods annually, or about 60 percent of the total  domestic tonnage shipped. Buses also play an important role, transporting  an estimated 631 million passengers annually. There are approximately  711,000 commercial motor carriers registered in MCMIS, about 9 million  trucks and buses, and more than 10 million drivers. Most motor carriers  are small; about 51 percent operate one vehicle, and another 31 percent  operate two to four vehicles. Carrier operations vary widely in size,  however, and some of the largest motor carriers operate upwards of  50,000 vehicles. Carriers continually enter and exit the industry. Since  1998, the industry has increased in size by an average of about 29,000  interstate carriers per year.
In the United States, commercial motor carriers account for less than 5  percent of all highway crashes, but these crashes result in about 13  percent of all highway deaths, or about 5,500 of the approximately 43,000  highway fatalities that occur nationwide annually. In addition, about  106,000 of the approximately 2.7 million highway injuries per year involve  motor carriers. The fatality rate for trucks has generally been decreasing  over the past 30 years, but this decrease has leveled off, and the rate has  been fairly stable since the mid-1990s. The fatality rate for buses has  improved slightly from 1975 to 2005 but has more annual variability than  the fatality rate for trucks due to a much smaller total vehicle miles  traveled. (See fig. 1.)
Congress created FMCSA through the Motor Carrier Safety Improvement  Act of 1999 to reduce crashes, injuries, and fatalities involving commercial  motor vehicles. To accomplish this mission, FMCSA carries out a number  of enforcement, education, and outreach activities. FMCSA uses  enforcement as its primary approach for reducing the number of crashes,  fatalities, and injuries involving trucks and buses. Some of FMCSA’s  enforcement programs include compliance reviews, which are on-site  reviews of carriers’ records and operations to determine compliance with  regulations; safety audits of new interstate carriers; and roadside  inspections of drivers and vehicles.
FMCSA’s education and outreach programs are intended to promote  motor carrier safety and consumer awareness. One of the programs is the  New Entrant program, which is designed to inform newly registered motor  carriers about motor carrier safety standards and regulations to help them  comply with FMCSA’s requirements. Other programs are designed to  identify unregistered carriers and get them to register, promote increased  safety belt use among commercial drivers, and inform organizations and  individuals that hire buses how to make safe choices. FMCSA plans to  make major revisions to its compliance and enforcement approach under  an initiative called Comprehensive Safety Analysis 2010.
Compliance reviews are an important enforcement tool because they  allow FMCSA to take an in-depth look at carriers that have been identified  as posing high crash risks because of high crash rates or poor safety  performance records. Motor carriers may be identified as high risk from  SafeStat or through calls to FMCSA’s complaint hotline. Carriers are given  a satisfactory, conditional, or unsatisfactory safety rating. A conditional  rating means the carrier is allowed to continue operating, but FMCSA may  schedule a follow-up compliance review to ensure that problems noted in  the first compliance review are addressed. An unsatisfactory rating must  be addressed or the carrier is placed out of service, meaning it is no longer  allowed to do business, and the carrier may face legal enforcement actions  undertaken by FMCSA. Compliance reviews can take several days to  complete, depending on the size of the carrier, and may result in  enforcement actions being taken against a carrier.
FMCSA uses both its own inspectors and state inspectors to carry out its  enforcement activities. In total, about 750 staff are available to perform  compliance reviews, and more than 10,000 staff do vehicle and driver  inspections at weigh stations and other points. Together, FMCSA and its  state partners perform about 16,000 compliance reviews a year, which  cover about 2 percent of the nation’s 711,000 carriers.
Because the number of inspectors is small compared with the size of the  motor carrier industry, FMCSA prioritizes carriers for compliance reviews.  To do so, it uses SafeStat to identify carriers that pose high crash risks.  SafeStat is a model that uses information gathered from crashes, roadside  inspections, traffic violations, compliance reviews, and enforcement cases  to determine a motor carrier’s safety performance relative to that of other  motor carriers that have similar exposure in these areas. A carrier’s score  is calculated on the basis of its performance in four safety evaluation  areas:    Accident safety evaluation area: The accident safety evaluation area  reflects a carrier’s crash history relative to other motor carriers’ histories.  The safety evaluation area is based on state-reported crash data, vehicle  data from MCMIS, and data on reportable crashes and annual vehicle  miles traveled from the most recent compliance review. A carrier must  have two or more reportable crashes within the last 30 months to have the  potential to receive a deficient value and thus be made a priority for a  compliance review.
Driver safety evaluation area: The driver safety evaluation area reflects a  carrier’s driver-related safety performance and compliance relative to  other motor carriers. The driver safety evaluation area is based on  violations cited in roadside inspections that have been performed within  the last 30 months and compliance reviews that have occurred within the  last 18 months, together with the number of drivers listed in MCMIS. A  carrier must have three or more driver inspections, three or more moving  violations, or at least one acute or critical violation of driver regulations  from a compliance review to have the potential to receive a deficient value  and thus be made a priority for a compliance review.
Vehicle safety evaluation area: The vehicle safety evaluation area reflects  a carrier’s vehicle-related safety performance and compliance relative to  other motor carriers. The vehicle safety evaluation area is based on  violations identified during vehicle roadside inspections that have  occurred within the last 30 months or vehicle-related acute and critical  violations of regulations discovered during compliance reviews that have  occurred within the last 18 months. A carrier must have either three or  more vehicle inspections or at least one acute or critical violation of  vehicle regulations from a compliance review to have the potential to  receive a deficient value and thus be made a priority for a compliance  review.
Safety management safety evaluation area: The safety management  safety evaluation area reflects a carrier’s safety management relative to  other motor carriers. It is based on the results of violations cited in closed  enforcement cases in the past 6 years or violations of regulations related  to hazardous materials and safety management discovered during a  compliance review performed within the last 18 months. A carrier must  have had at least one enforcement case initiated and closed or at least two  enforcement cases closed within the past 6 years, or at least one acute,  critical, or severe violation of hazardous material or safety management  regulations identified during a compliance review within the last 18  months to have the potential to receive a deficient value and thus be made  a priority for a compliance review.
A motor carrier’s score is based on its relative ranking, indicated as a  value, in each of the four safety evaluation areas. For example, if a carrier  receives a value of 75 in the accident safety evaluation area, then 75  percent of all carriers with sufficient data for evaluation performed better  in that safety evaluation area, while 25 percent performed worse. The  calculation used to determine a motor carrier’s SafeStat score is as  follows:  SafeStat Score = (2.0x accident value) + (1.5x driver value)  + vehicle value + safety management value  As shown in the formula, the accident and driver safety evaluation areas  have 2.0 and 1.5 times the weight, respectively, of the vehicle and safety  management safety evaluation areas. Safety evaluation area values less  than 75 are ignored in the formula used to determine the SafeStat score.  For example, a carrier with values of 74 for all four safety evaluation areas  has a SafeStat score of 0. FMCSA assigned more weight to these safety  evaluation areas because, according to FMCSA, crashes and driver  violations correlate relatively better with future crash risk. In addition,  more weight is assigned to fatal crashes and to crashes that occurred  within the last 18 months. In consultation with state transportation  officials, insurance industry representatives, safety advocates, and the  motor carrier industry, FMCSA used its expert judgment and professional  knowledge to assign these weights, rather than determining them through  a statistical approach, such as regression modeling.
FMCSA assigns carriers categories ranging from A to H according to their  performance in each of the safety evaluation areas. A carrier is considered  to be deficient in a safety evaluation area if it receives a value of 75 or  higher in that particular safety evaluation area. Although a carrier may  receive a value in any of the four safety evaluation areas, the carrier  receives a SafeStat score only if it is deficient in one or more safety  evaluation areas. Carriers that are deficient in two or more safety  evaluation areas and have a SafeStat score of 225 or more are considered  to pose high crash risks and are placed in category A or B. (See table 1.)  Carriers that are deficient in two safety evaluation areas but have a  SafeStat score of less than 225 are placed in category C and receive a  medium priority for compliance reviews. Carriers that are deficient in only  one of the safety evaluation areas are placed in category D, E, F, or G.  Carriers that are not deficient in any of the safety evaluation areas do not  receive a SafeStat score and are placed in category H.
Of the 622,000 motor carriers listed in MCMIS as having one or more  vehicles in June 2004, about 140,000, or 23 percent, received a SafeStat  category A through H. There are several reasons why a small proportion of  carriers receive a score. First, approximately 305,900, or about 42 percent,  of the carriers have crash, vehicle inspection, driver inspection, or  enforcement data of any kind. SafeStat relies on these data to calculate a  motor carrier’s score, so carriers without such data are not rated by  SafeStat. It is likely that some of the carriers listed in MCMIS are no longer  in business, but it is also possible that these carriers had no crashes,  inspections, or compliance reviews in the 30-month period prior to June  2004. Second, a carrier must meet the minimum requirements to be  assigned a value in a given safety evaluation area. If, for example, a  carrier had only one reportable crash within the last 30 months, then the  carrier would not be assigned an accident safety evaluation area value. Of  the 305,900 carriers that have any safety data in SafeStat, 140,000 met the  SafeStat minimum requirements in one or more safety evaluation areas. Of  these 140,000 carriers, 45,000 were rated in categories A through G. The  other carriers were placed in category H because they were not  considered deficient, meaning they did not receive a value of 75 or more in  any of the safety evaluation areas.
The design of SafeStat and its data sufficiency requirements increase the  likelihood that larger motor carriers will be deficient in one of the safety  evaluation areas, in other words, rated in categories A through G, than are  small carriers. About 51 percent of all carriers listed in MCMIS operate one  vehicle, and about 3 percent of them received a SafeStat rating in  categories A through G. (See table 2.) In contrast, fewer than 1 percent of  the carriers listed in MCMIS have more than 100 vehicles, and nearly 25  percent of them received a SafeStat rating in categories A through G.
section_title: A Statistical Approach Would Better Identify Carriers That Pose High Crash Risks Than Does FMCSA’s Current Approach
paragraphs: We found that FMCSA could improve SafeStat’s ability to identify carriers  that pose high crash risks if it applied a statistical approach, called a  negative binomial regression model, to the four SafeStat safety evaluation  areas instead of its current approach. Through this change, FMCSA could  more efficiently target compliance reviews to the set of carriers that pose  the greatest crash risk. Applying a negative binomial regression model  would improve the identification of high risk carriers over SafeStat’s  performance by about 9 percent, compared with the current approach,  which incorporates safety data weighted in accordance with the  professional judgment and experience of SafeStat’s designers. Moreover,  according to our analysis, this 9 percent improvement would enable  FMCSA to identify carriers with almost twice as many crashes in the  following 18 months as those carriers identified under its current  approach. Targeting these high-risk carriers would result in FMCSA giving  compliance reviews to carriers that experienced both a higher crash rate  and, in conjunction with the higher crash rate, 9,500 more crashes over an  18-month period than those identified by the SafeStat model. Applying a  negative binomial regression model approach to the SafeStat safety  evaluation areas would be easy to implement and, in our opinion, would  be consistent with other FMCSA uses for SafeStat beyond identifying  carriers that pose high risks for crashes. In addition, adopting a negative  binomial regression model approach would be beneficial even if FMCSA  makes major revisions to its compliance and enforcement approach in the  coming years under its Comprehensive Safety Analysis 2010 initiative.  Overall, other changes to the SafeStat model that we explored, such as  modifying decision rules used in the construction of the safety evaluation  areas, did not improve the model’s overall performance.
subsections: section_title: Regression Models Identify Carriers That Pose High Crash Risks Better Than Expert Judgment
paragraphs: Although SafeStat is nearly twice as effective as (83 percent better than)  random selection in identifying carriers that pose high crash risks and,  therefore, has value for improving safety, we found that FMCSA could  improve SafeStat’s ability to identify such carriers by about 9 percent if it  applied a negative binomial regression model approach to its analysis of  motor carrier safety data. The use of a regression model does not entail  assigning the letter categories currently assigned by the SafeStat model.  Rather, the model predicts carriers’ crash risks, sorts the carriers  according to their risk level, and assigns a high priority for a compliance  review to the highest risk carriers. The improvement in identification of  high-risk carriers, which we observed with the negative binomial  regression model, is consistent with results obtained in an earlier analysis  of MCMIS data performed by a team of researchers at Oak Ridge National  Laboratory.
To compare the effectiveness of regression models and SafeStat in  identifying carriers that pose high crash risks, we applied several  regression models to the four safety evaluation areas (accident, driver,  vehicle, and safety management) used by the SafeStat model. We  recalculated SafeStat’s June 2004 accident safety evaluation area values  because the data FMCSA provided on the number of crashes for each  carrier differed in 2006 from the data used in the model in 2004. Using our  accident safety evaluation area value and the original driver, vehicle, and  safety management safety evaluation area values from June 2004, we  selected the 4,989 carriers that our regression models identified as the  highest crash risks, calculated the crash rate per 1,000 vehicles for these  carriers over the next 18 months, and compared this rate with the crash  rate per 1,000 vehicles for the 4,989 carriers identified by the SafeStat  model as posing high crash risks (categories A and B).
All of the regression models that we estimated were at least as effective as  SafeStat in identifying motor carriers that posed high crash risks. (See app.  III for these results.) Of these, the negative binomial regression approach  gave the best results and proved 9 percent more effective than SafeStat, as  measured by future crashes per 1,000 vehicles. The set of carriers in  SafeStat categories A and B had a crash rate of 102 per 1,000 vehicles for  the 18 months after June 2004 while the set of high-risk carriers identified  by the negative binomial regression model had 111 crashes per 1,000  vehicles. Even though this 9 percent improvement rate seems modest, it  translates into nearly twice as many “future crashes” identified.  Specifically, the negative binomial regression model identified carriers  that had nearly twice as many crashes (from July 2004 to December 2005)  as the carriers identified by SafeStat—19,580 crashes compared with  10,076.
SafeStat (categories A and B) and our negative binomial regression model  identified many of the same carriers—1,924 of the 4,989 (39 percent)—as  posing high crash risks. However, our model also identified a number of  high-risk carriers that SafeStat did not identify, and vice versa. For  example, our model identified 2,244 carriers as posing high crash risks,  while SafeStat placed these carriers in category D (the accident area),  assigning them a lower priority for compliance reviews. One reason for  this difference is the decision rules that SafeStat employs. Under SafeStat,  carriers must perform worse than 75 percent of all carriers to be  considered deficient in any safety evaluation area. The regression  approach identifies the carriers with the highest crash risks regardless of  how they compare with their peers in individual areas. For example, we  identified as posing high crash risks 482 carriers that SafeStat did not  consider at all for compliance reviews because the carriers had not  performed worse than 75 percent of their peers in any of the four safety  evaluation areas.
section_title: FMCSA Can Apply a Regression Model Approach in the Short Term, Even Though It Is Planning to Overhaul SafeStat
paragraphs: In the short term, FMCSA could easily implement a regression model  approach for SafeStat. All the information required as input for the  negative binomial regression model is already entered into SafeStat. In  addition, a standard statistical package can be used to apply the negative  binomial approach to the four SafeStat safety evaluation areas. Like  SafeStat, the negative binomial regression model would be run every  month to produce a list of motor carriers that pose high crash risks, and  these carriers would then be assigned priorities for a compliance review.  As with SafeStat, the results of the negative binomial model would change  slightly each month with the addition of new safety data to MCMIS.
In discussing the concept of adopting a negative binomial regression  model approach with FMCSA officials, they were interested in  understanding how the use of the negative binomial regression model  results could be used to identify and improve the safety of those carriers  that pose the greatest crash risks (much as the SafeStat categories of A  and B do now) and how it could employ the proposed approach for  current uses beyond identifying carriers that pose high crash risks. These  uses include providing an understandable public display to shippers,  insurers, and others who are interested in the safety of carriers; selecting  carriers for roadside inspections; and trying to gain carriers’ compliance  with driver and vehicle safety rules, when these carriers may not have  crashes, consistent with agency efforts.
Identifying and improving the safety of carriers that pose high crash  risks. The negative binomial regression model approach would produce a  rank order listing of carriers by crash risk and by the predicted number of  crashes. For compliance reviews, FMCSA could choose those carriers with  the greatest number of predicted crashes. FMCSA would choose the  number of carriers to review based on the resources available to it, much  as it currently does.
Regarding improving the safety of carriers that pose high crash risks,  FMCSA currently enrolls carriers that receive a SafeStat category of A, B,  or C in the Motor Carrier Safety Improvement Program. This program aims  to improve the safety of high-risk carriers through (1) a repetitive cycle of  identification, data gathering, and assessment and (2) progressively  harsher treatments applied to carriers that do not improve their safety.  The use of a negative binomial regression model would not affect the  structure or workings of this program, other than to better identify carriers  that pose high crash risks. As discussed above, FMCSA would use the  regression model’s results to identify the highest risk carriers and then  intervene using its existing approaches (such as issuing warning letters,  conducting follow-up compliance reviews, or levying civil penalties) as  treatment.
Providing an understandable display to the public. FMCSA could choose  to provide a rank order listing of carriers together with the associated  number of predicted crashes or it could look for natural breaks in the  predicted number of crashes and associate a category—such as “category  A” to these carriers.
Selecting carriers for roadside inspections. Safety rankings from the  SafeStat model are also used in FMCSA’s Inspection Selection System to  prioritize carriers for roadside driver and vehicle inspections. The negative  binomial regression model optimizes the identification of carriers by crash  risk using safety evaluation area information. The negative binomial  regression model approach that we describe in this report retains  SafeStat’s basic design with four safety management areas (driver, vehicle,  accident, and safety management). Therefore, FMCSA could use the  negative binomial regression model results to identify carriers that pose a  high crash risk, the results from the driver and vehicle safety evaluation  areas, or both, to target carriers or vehicles for roadside driver and vehicle  inspections.
Furthering agency efforts to gain compliance with driver and vehicle  safety rules for carriers that do not experience crashes (or a sufficient  number of crashes to pose a high risk for crashes). FMCSA was  interested in understanding how, if at all, the negative binomial regression  model approach would affect gaining compliance against carriers that may  routinely violate safety rules (such as drivers’ hours of service  requirements), but where these violations do not lead to crashes. As  discussed above, the negative binomial regression model approach retains  SafeStat’s four safety evaluation areas. Where it differs, is that it assigns  different weights to those areas based on a statistical procedure, rather  than having the weights assigned by expert judgment. As a result, FMCSA  would still be able to identify carriers with many driver, vehicle, and safety  management violations.
Other opportunities also exist for FMCSA to improve the ability of  regression models to identify carriers that pose high crash risks. In 2005, a  FMCSA compliance review work group reported a positive correlation  between driver hours of service violations and crash rates. Because  FMCSA can link violations of specific regulatory provisions, including  those limiting driver hours of service, to the crash experience of the  carriers involved, it has the opportunity to improve the violation severity  weighting used in constructing the driver and vehicle safety evaluation  areas. FMCSA has detailed violation data from roadside inspections and  can statistically analyze these data to find other strong relationships with  carriers’ crash risks. Changes made to the safety evaluation area  methodology to strengthen the association with crash risk will improve  the ability of the negative binomial regression model to identify carriers  that pose high crash risks.
FMCSA has expressed doubts in the past when analysts have proposed  switching to a regression model approach. For example, Oak Ridge  National Laboratory advocated using a regression model approach in place  of SafeStat in 2004, but FMCSA was reluctant to move away from its  expert judgment model because it believed that the regression model  approach would place undue weight on the accident safety evaluation area  in determining priorities for compliance reviews, thereby diminishing the  incentive for motor carriers to comply with the many safety regulations  that feed into the driver, vehicle, and safety management safety evaluation  areas. In FMCSA’s view, carriers would be less likely to comply with these  regulations because violations in the driver, vehicle, and safety  management areas would be less likely to lead to compliance reviews  under a regression model approach that placed a heavy emphasis on  crashes. Our view is that adopting a negative binomial regression model  approach would better identify carriers that pose high crash risks and  would thus further FMCSA’s primary mission of ensuring safe operating  practices among commercial interstate motor carriers.
Over the longer term, FMCSA is considering a complete overhaul of its  safety fitness determinations with its Comprehensive Safety Analysis 2010  initiative. This planned comprehensive review and analysis of the agency’s  compliance and enforcement programs may result in a new operational  model for identifying drivers and carriers that pose safety problems and  for intervening to address those problems. FMCSA expects to deploy the  results of this initiative in 2010. In our opinion, given the relative ease of  adopting the regression modeling approach discussed in this report, and  the immediate benefits that can be achieved, there is no reason to wait for  FMCSA to complete its initiative, even if the initiative results in major  revisions to the SafeStat model.
section_title: Modifications of SafeStat Did Not Improve Crash Identification
paragraphs: Besides investigating whether the use of regression models could improve  SafeStat’s ability to identify carriers that pose high crash risks, we  explored whether the existing model could be improved by changing  several of its decision rules. Overall, these changes did not enhance the  model’s ability to identify carriers that pose high crash risks. As long as  FMCSA continues to estimate the safety evaluation area values with its  present methodology, the rules we investigated help make the  identification of high-risk motor carriers more efficient for both SafeStat  and the negative binomial regression model.
Because the SafeStat model is composed of many components, we  selected three decision rules for analysis. We chose these three rules  because they are important pillars of the SafeStat model’s methodology for  constructing the safety evaluation areas and because we could complete  our analysis of them during the time we had to perform our work. A fuller  exploration of areas with high potential to improve the identification of  carriers that pose high crash risks would be a long-term effort, and FMCSA  plans to address this work as part of the Comprehensive Safety Analysis  2010 initiative.
Removing comparison groups. As part of its methodology for calculating  the accident, driver, and vehicle safety evaluation area values, SafeStat  divides carriers into comparison groups. For example, in the driver safety  evaluation area, SafeStat groups carriers by the number of moving  violations they have, placing them in one of four groups (3 to 9, 10 to 28, 29  to 94, and 95 or more). SafeStat uses the comparison groups to control  for the size of the carrier. We removed all the comparison groups in each  of the three safety evaluation areas, recalculated their values, and  compared the number of crashes in which the carriers were involved and  their crash rates, for each of the SafeStat categories A through H, with the  SafeStat results in which comparison groups were retained.
Removing minimum event requirements. SafeStat imposes minimum  event requirements. For example, as noted, SafeStat does not consider a  carrier’s moving violations if, in the aggregate, its drivers had fewer than  three moving violations over a 30-month period. FMCSA does not calculate  a safety evaluation area value for carriers with fewer than three events in  an attempt to control for carriers that have infrequent, rather than possibly  systemic, safety problems. We removed the requirement to have a  minimum number of events (such as moving violations, crashes, and  inspections), recalculated the three safety evaluation values, and  compared the number of crashes in which the carriers were involved and  their crash rates, for each of the SafeStat categories A through H, with the  SafeStat results in which minimum event requirements were retained.
Removing time and severity weights. The SafeStat formula weights more  recent events and more severe events more heavily than less recent or less  severe events in the accident, driver, and vehicle safety evaluation areas.  For example, the results of vehicle roadside inspections performed within  the latest 6 months receive three times the weight of inspections  performed 2 years ago. Similarly, crashes involving deaths or injuries  receive twice as much weight as those that resulted in property damage  only. We removed the time and severity weights for the three safety  evaluation areas, recalculated these values, and compared the number of  crashes in which the carriers were involved and their crash rates, for each  of the SafeStat categories A through H, with the SafeStat results in which  time and severity weights were retained.
Simultaneous changes to comparison group, event, and time severity  requirements. Finally, we simultaneously removed comparison groups,  minimum event requirements, and time and severity weights and  compared the number of crashes in which the carriers were involved and  their crash rates, for each of the SafeStat categories A through H, with the  SafeStat results in which comparison groups, minimum event  requirements, and time and severity weights were retained.
The results of each of our individual analyses and of making all changes  simultaneously produced one of two outcomes, neither of which was  considered more desirable. Relaxing the minimum data requirements  greatly increased the number of carriers identified as high risk without  increasing the overall number of predicted crashes over the subsequent 18  months, thus reducing the effectiveness of the SafeStat model. Removing  comparison groups and removing time and severity weights had the effect  of reducing the future crashes per 1,000 vehicles among those carriers  identified as high risk, also reducing the effectiveness of the SafeStat  model. As a result, we are not reporting on these results in detail. Trying to  modify the decision rules used in SafeStat did highlight the balance that  FMCSA has to strike between maximizing the identification of companies  with the largest number of crashes (usually larger carriers) and those  carriers with the greatest safety risk (which can be of any size).
section_title: Despite Quality Problems, FMCSA’s Crash Data Can Be Used to Compare Methods for Identifying Carriers That Pose High Crash Risks
paragraphs: The quality of crash data is a long-standing problem that potentially  hindered FMCSA’s ability to accurately identify carriers that pose high  crash risks. Despite the problems of late-reported crashes and incomplete  and inaccurate data on crashes during the period we studied, we  determined that the data were of sufficient quality for our use, which was  to assess how the application of regression models might improve the  ability to identify high-risk carriers over the current approach—not to  determine absolute measures of crash risk. Our reasoning is based on the  fact that we used the same data set to compare the results of the SafeStat  model and the regression models. Limitations in the data would apply  equally to both results. FMCSA has recently undertaken a number of  efforts to improve crash data quality.
subsections: section_title: Late Reporting Had a Small Effect on SafeStat’s Ability to Identify High- Risk Carriers
paragraphs: FMCSA’s guidance provides that states report all crashes to MCMIS within  90 days of their occurrence. Late reporting can cause SafeStat to miss  some of the carriers that should have received a SafeStat score.  Alternatively, since SafeStat’s scoring involves a relative ranking of  carriers, a carrier may receive a SafeStat score and have to undergo a  compliance review because crash data for a higher risk carrier were  reported late and not included in the calculation.
Late reporting affected SafeStat’s ability to identify all high-risk carriers to  a small degree—about 6 percent-—for the period that we studied. Late  reporting of crashes by states affected the safety rankings of more than  600 carriers, both positively and negatively. When SafeStat analyzed the  2004 data, which did not include the late-reported crashes, it identified  4,989 motor carriers as highest risk, meaning they received a category A or  B ranking. With the addition of late-reported crashes, 481 carriers moved  into the highest risk category, and 182 carriers dropped out of the highest  risk category, resulting in a net increase of 299 carriers (6 percent) in the  highest risk category. After the late-reported crashes were added, 481  carriers that originally received a category C, D, E, F, or G SafeStat rating  received an A or B rating. These carriers would not originally have been  given a high priority for a compliance review because the SafeStat  calculation did not take into account all of their crashes. On the other  hand, a small number of carriers would have received a lower priority if  the late-reported crashes had been included in their score. Specifically,  182 carriers – or fewer than 4 percent of those ranked, fell from the A or B  category into the C, D, E, F, or G category once the late-reported crashes  were included. These carriers would not have been considered high  priority for a compliance review if all crashes had been reported on time.  This does not have a big effect on the overall motor carrier population,  however, as only 4 percent of carriers originally identified as highest risk  were negatively affected by late reporting.
The timeliness of crash reporting has shown steady and marked  improvement. The median number of days it took states to report crashes  to MCMIS dropped from 225 days in calendar year 2001 to 57 days in 2005  (the latest data available at the time of our analysis). In addition, the  percentage of crashes reported by states within 90 days of occurrence has  jumped from 32 percent in fiscal year 2000 to 89 percent in fiscal year  2006. (See fig. 2.)
section_title: Incomplete Data from States Potentially Limit SafeStat’s Identification of All Carriers That Pose High Crash Risks
paragraphs: FMCSA uses a motor carrier identification number, which is unique to  each carrier, as the primary means of linking inspections, crashes, and  compliance reviews to motor carriers. Approximately 184,000 (76 percent)  of the 244,000 crashes reported to MCMIS between December 2001 and  June 2004 involved interstate carriers. Of these 184,000 crashes, nearly  24,000 (13 percent) were missing this identification number. As a result,  FMCSA could not match these crashes to motor carriers or use them in  SafeStat. In addition, the carrier identification number could not be  matched to a number listed in MCMIS for 15,000 (8 percent) other crashes  that involved interstate carriers. Missing data or being unable to match  data for nearly one quarter of the crashes during the period of our review  potentially has a large impact on a motor carrier’s SafeStat score because  SafeStat treats crashes as the most important source of information for  assessing motor carrier crash risk. Theoretically, information exists to  match crash records to motor carriers by other means, but such matching  would require too much manual work to be practicable.
We were not able to quantify the actual effect of either the missing data or  the data that could not be matched for MCMIS overall. To do so would  have required us to gather crash records at the state level—an effort that  was impractical. For the same reason, we cannot quantify the effects of  FMCSA’s efforts to improve the completeness of the data (discussed  later). However, a series of reports by the University of Michigan  Transportation Research Institute sheds some light on the completeness of  the data submitted to MCMIS by the states. One of the goals of the  research was to determine the states’ crash reporting rates. Reporting  rates varied greatly among the 14 states studied, ranging from 9 percent in  New Mexico in 2003 to 87 percent in Nebraska in 2005. It is not possible to  draw wide-scale conclusions about whether state reporting rates are  improving over time because only two of the states—Missouri and Ohio—- were studied in multiple years. However, in these two states, the reporting  rate did improve. Missouri experienced a large improvement in its  reporting rate, with 61 percent of eligible crashes reported in 2001, and 83  percent reported in 2005. Ohio’s improvement was more modest,  increasing from 39 percent in 2000 to 43 percent in 2005.
The University of Michigan Transportation Research Institute’s reports  also identified a number of factors that may affect states’ reporting rates.  One of the main factors affecting reporting rates is the reporting officer’s  understanding of crash reporting requirements. The studies note that  reporting rates are generally lower for less serious crashes and for crashes  involving smaller vehicles, which may indicate that there is some  confusion about which crashes are reportable. Some states, such as  Missouri, aid the officer by explicitly listing reporting criteria on the police  accident reporting form, while other states, such as Washington, leave it  up to the officer to complete certain sections of the form if the crash is  reportable, but the form includes no guidance on reportable crashes. Yet  other states, such as North Carolina and Illinois, have taken this task out  of officers’ hands and include all reporting elements on the police accident  reporting form. Reportable crashes are then selected centrally by the state,  and the required data are transmitted to MCMIS.
section_title: Inaccurate Data Potentially Limit SafeStat’s Ability to Identify Carriers That Pose High Crash Risks
paragraphs: Inaccurate data, such as reporting a nonqualifying crash to FMCSA,  potentially has a large impact on a motor carrier’s SafeStat score because  SafeStat treats crashes as the most important source of information for  assessing motor carrier crash risk. For the same reasons as discussed in  the preceding section, we were neither able to quantify these effects nor  determine how data accuracy has improved for MCMIS overall.
The University of Michigan Transportation Research Institute’s reports on  crash reporting show that, among the 14 states studied, incorrect reporting  of crash data is widespread. In recent reports, the researchers found that,  in 2005, Ohio incorrectly reported 1,094 (22 percent) of the 5,037 cases,  and Louisiana incorrectly reported 137 (5 percent) of the 2,699 cases. In  Ohio, most of the incorrectly reported crashes did not qualify because they  did not meet the crash severity threshold. In contrast, most of the  incorrectly reported crashes in Louisiana did not qualify because they did  not involve vehicles eligible for reporting. Other states studied by the  institute had similar problems with reporting crashes that did not meet the  criteria for reporting to MCMIS. These additional crashes could cause  some carriers to exceed the minimum number of crashes required to  receive a SafeStat rating and result in SafeStat’s mistakenly identifying  carriers as posing high crash risks. Because each report focuses on  reporting in one state in a particular year, it is not possible to identify the  number of cases that have been incorrectly reported nationwide and,  therefore, it is not possible to determine the impact of inaccurate reporting  on SafeStat’s calculations.
As noted in the University of Michigan Transportation Research Institute’s  reports, states may be unintentionally submitting incorrect data to MCMIS  because of difficulties in determining whether a crash meets the reporting  criteria. For example, in Missouri, pickups are systematically excluded  from MCMIS crash reporting, which may cause the state to miss reportable  crashes. However, some pickups may have vehicle weights above the  reporting threshold, making crashes involving them eligible for reporting.  There is no way for the state to determine which crashes involving  pickups qualify for reporting without examining the characteristics of each  vehicle. In this case, the number of omissions is likely to be relatively  small, but this example demonstrates the difficulty states may face when  identifying reportable crashes.
In addition, in some states, the information contained in the police  accident report may not be sufficient for the state to determine if a crash  meets the accident severity threshold. It is generally straightforward to  determine whether a fatality occurred as a result of a crash, but it may be  difficult to determine whether an injured person was transported for  medical attention or a vehicle was towed because of disabling damage. In  some states, such as Illinois and New Jersey, an officer can indicate on the  form if a vehicle was towed by checking a box, but there is no way to  identify whether the reason for towing was disabling damage. It is likely  that such uncertainty results in overreporting because some vehicles may  be towed for other reasons.
section_title: FMCSA Has Undertaken Efforts to Improve Crash Data Quality
paragraphs: FMCSA has taken steps to try and improve the quality of crash data  reporting. As we noted in November 2005, FMCSA has undertaken two  major efforts to help states improve the quality of crash data. One  program, the Safety Data Improvement Program, has provided funding to  states to implement or expand activities designed to improve the  completeness, timeliness, accuracy, and consistency of their data. FMCSA  has also used a data quality rating system to rate and display ratings for  states’ crash and inspection data quality. Due to its public nature, this map  serves as an incentive for states to make improvements in their data  quality.
To further improve these programs, FMCSA has made additional grants  available to states and implemented our recommendations to (1) establish  specific guidelines for assessing states’ requests for funding to support  data improvement in order to better assess and prioritize the requests and  (2) increase the usefulness of its state data quality map as a tool for  monitoring and measuring commercial motor vehicle crash data by  ensuring that the map adequately reflects the condition of the states’  commercial motor vehicle crash data.
In February 2004, FMCSA implemented Data Q’s, an online system that  allows for challenging and correcting erroneous crash or inspection data.  Users of this system include motor carriers, the general public, state  officials, and FMCSA. In addition, in response to a recent recommendation  by the Department of Transportation Inspector General, FMCSA is  planning to conduct a number of evaluations of the effectiveness of a  training course on crash data collection that it will be providing to states  by September 2008.
While the quality of crash reporting is sufficient for use in identifying  motor carriers that pose high crash risks and has started to improve,  commercial motor vehicle crash data continue to have some problems  with timeliness, completeness, and accuracy. These problems have been  well-documented in several studies, and FMCSA is taking steps to address  the problems through studies of each state’s crash reporting system and  grants to states to fund improvements. As a result, we are not making any  recommendations in this area.
section_title: Conclusion
paragraphs: Interstate commerce involving large trucks and buses has been growing  substantially, and this growth is expected to continue. While the number  of fatalities per million vehicle miles traveled has generally decreased over  the last 30 years, the fatality rate has leveled off and remained fairly steady  since the mid-1990s. FMCSA could more effectively address fatalities due  to crashes involving a commercial motor vehicle if it better targeted  compliance reviews to those carriers that pose the greatest crash risks.  Using a negative binomial regression model would further FMCSA’s  mission of reducing crashes through the more effective targeting of  compliance reviews to the set of carriers that pose the greatest crash risks.  In light of possible changes to FMCSA’s safety fitness determinations  resulting from its Comprehensive Safety Analysis 2010 initiative, we are  not suggesting that FMCSA undertake a complete and thorough  investigation of SafeStat. Rather, we are advocating that FMCSA apply a  statistical approach that employs the negative binomial regression model  rather than relying on the current SafeStat formula that was determined  through expert judgment. In our view, the substitution of a statistically  based approach would likely yield a markedly better ability to identify  carriers that pose high crash risks with relatively little time or effort on  FMCSA’s part.
section_title: Recommendation for Executive Action
paragraphs: We recommend that the Secretary of Transportation direct the  Administrator of FMCSA to apply a negative binomial regression model,  such as the one discussed in this report, to enhance the current SafeStat  methodology.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Department of Transportation for  its review and comment. In response, departmental officials, including  FMCSA’s Director of the Office of Enforcement and Compliance and  Director of the Office of Research and Analysis, noted that our report  provided useful insights and offered a potential avenue for further  improving the effectiveness of FMCSA’s efforts to reduce crashes  involving motor carriers. The agency indicated that it is already working to  improve upon SafeStat as part of its Comprehensive Safety Analysis 2010  initiative. FMCSA agreed that it would be useful for it to consider whether  there are both short and longer term measures that would incorporate the  type of analysis identified in our report, as an adjunct to the SafeStat  model, in order to better target compliance reviews so as to make the best  use of FMCSA’s resources to reduce crashes.
The agency expressed some concerns with the negative binomial  regression analysis, noting that its intent is to effectively target its  compliance activities based on a broader range of factors than is  considered in the negative binomial regression analysis approach  described in our draft report, which increases reliance on past crashes as a  predictor of future crashes while apparently de-emphasizing known driver,  vehicle, or safety management compliance issues. FMCSA told us that it  incorporates a broad range of information including driver behavior,  vehicle condition, and safety management in an attempt to capture and  enable the agency to act on accident precursors in order to reduce  crashes.
FMCSA is correct in concluding that the use of the negative binomial  regression approach could tilt enforcement heavily toward carriers that  have experienced crashes and away from other aspects of its problem  areas, such as violation of vehicle safety standards, that are intended to  prevent crashes. That is because the present SafeStat model does not  statistically assign weights to the accident, driver, vehicle, and safety  management areas. In addition, the negative binomial regression approach  fully considers information on the results of driver and vehicle inspection  data and safety management data. We used the same data that FMCSA  used, with some adjustments as new information became available. While  we found that the driver, vehicle, and safety management evaluation area  scores are correlated with the future crash risk of a carrier, the accident  evaluation area correlates the most with future crash risk. We recognize  that FMCSA selects carriers for compliance reviews for multiple reasons,  such as to respond to complaints, and we would expect that it would  retain this flexibility if it adopted the negative binomial regression  approach.
FMCSA also indicated that greater reliance on crash data increases  emphasis on the least reliable available data set, and one that is out of the  organization’s direct control—crash reporting. While our draft report  found that crash reporting has improved, and that late reporting has not  significantly impaired FMCSA’s use of the SafeStat model, FMCSA noted  that the reliance on previous crashes in the negative binomial regression  analysis described in our draft report could result in greater sensitivity to  the crash data quality issues.
As FMCSA noted in its comments, our results showed that the effect of  late-reported data was minimal. Also, as mentioned in our draft report and  in this final report, it was not practical to determine the effect, if any, on  SafeStat rankings of correcting inaccurate data or adding incomplete data.  Since June 2004, FMCSA has devoted considerable efforts to improving  the quality of the crash data it receives from the states. States are now  tracked quarterly for the completeness, timeliness, and accuracy of their  crash reporting. As FMCSA continues its efforts to have states improve  these data, any sensitivity of results to crash data quality issues for the  negative binomial regression approach should diminish.
We are sending copies of this report to congressional committees and  subcommittees with responsibility for surface transportation safety issues;  the Secretary of Transportation; the Administrator, FMCSA; and the  Director, Office of Management and Budget. We also will make copies  available to others upon request. In addition, this report will be available  at no charge on the GAO Web site at http://www.gao.gov.
If you have any questions about this report, please either contact Sidney H.  Schwartz at (202) 512-7387 or Susan A. Fleming at (202) 512-2834.  Alternatively, they may be reached at schwartzsh@gao.gov or  flemings@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Staff who made key contributions to this report are Carl Barden, Elizabeth  Eisenstadt, Laurie Hamilton, Lisa Mirel, Stephanie Purcell, and James  Ratzenberger.
section_title: Appendix I: Results of Other Assessments of the SafeStat Model’s Ability to Identify Motor Carriers That Pose High Crash Risks
paragraphs: Several studies by the Volpe National Transportation Systems Center  (Volpe), the Department of Transportation’s Office of Inspector General,  the Oak Ridge National Laboratory (Oak Ridge), and others have assessed  the predictive capability of the Motor Carrier Safety Status Measurement  System (SafeStat) model and the data used by that model. In general, those  studies that assessed the predictive power of SafeStat offered suggestions  to increase that power, and those studies that assessed data quality found  weaknesses in the data that the Federal Motor Carrier Safety  Administration (FMCSA) relies upon.
subsections: section_title: Assessments of SafeStat’s Predictive Capability
paragraphs: The studies we reviewed covered topics such as comparing SafeStat with  random selection to determine which does a better job of selecting  carriers that pose high crash risks, assessing whether statistical  approaches could improve that selection, and analyzing whether carrier  financial positions or driver convictions are associated with crash risk.
subsections: section_title: Predictive Capability of SafeStat Compared with Random Selection
paragraphs: In studies of the SafeStat model published in 2004 and 1998, Volpe  analyzed retrospective data to determine how many crashes the carriers in  SafeStat categories A and B experienced over the following 18 months.  The 2004 study used the carrier rankings generated by the SafeStat model  on March 24, 2001. Volpe then compared the SafeStat carrier safety ratings  with state-reported data on crashes that occurred between March 25, 2001,  and September 24, 2002, to assess the model’s performance. For each  carrier, Volpe calculated a total number of crashes, weighted for time and  severity, and then estimated a crash rate per 1,000 vehicles for comparing  carriers in SafeStat categories A and B with the carriers in other SafeStat  categories. The 1998 Volpe study used a similar methodology. Each study  used a constrained subset of carriers rather than the full list contained in  the Motor Carrier Management Information System (MCMIS). Both  studies found that the crash rate for the carriers in SafeStat categories A  and B was substantially higher than the other carriers during the 18  months after the respective SafeStat run. On the basis of this finding,  Volpe concluded that the SafeStat model worked.
In response to a recommendation by the Department of Transportation’s  Office of Inspector General, FMCSA contracted with Oak Ridge to  independently review the SafeStat model. Oak Ridge assessed the SafeStat  model’s performance and used the same data set (for March 24, 2001),  provided by Volpe, that Volpe had used in its 2004 evaluation. Perhaps not  surprisingly, Oak Ridge obtained a similar result for the weighted crash  rate of carriers in SafeStat categories A and B over the 18-month follow-up  period. As with the Volpe study, the Oak Ridge study was constrained  because it was based on a limited data set rather than the entire MCMIS  data set.
section_title: Application of Regression Models to Safety Data
paragraphs: While SafeStat does better than simple random selection in identifying  carriers that pose high crash risks, other methods can also be used to  achieve this outcome. Oak Ridge extended Volpe’s analysis by applying  regression models to identify carriers that pose high crash risks.  Specifically, Oak Ridge applied a Poisson regression model and a negative  binomial model using the safety evaluation area values as independent  variables to a weighted count of crashes that occurred in the 30 months  before March 24, 2001. (For more information on statistical analyses, see  app. III.)
In addition, Oak Ridge applied the empirical Bayes method to the negative  binomial regression model and assessed the variability of carrier crash  counts by estimating confidence intervals. Oak Ridge found that the  negative binomial model worked well in identifying carriers that pose high  crash risks. However, the data set Oak Ridge used did not include any  carriers with one reported crash in the 30 months before March 24, 2001.  Because data included only carriers with zero or two or more reported  crashes, the distribution of crashes was truncated.
Since the Oak Ridge regression model analysis did not cover carriers with  safety evaluation area data and one reported crash, the findings from the  study are limited in their generalizability. However, other analyses of  crashes at intersections and on road segments have also found that the  negative binomial regression model works well. In addition, our analysis  using a more recent and more comprehensive data set supports the finding  that the negative binomial regression model performs better than the  SafeStat model.
The studies carried out by other authors advocate the use of the empirical  Bayes method in conjunction with a negative binomial regression model to  estimate crash risk. Oak Ridge also applied this model to identify motor  carriers that pose high crash risks. We applied this method to the 2004  SafeStat data and found that the empirical Bayes method best identified  the carriers with the largest number of crashes in the 18 months after June  25, 2004. However, the crash rate per 1,000 vehicles was much lower than  that for carriers in SafeStat categories A and B. We analyzed this result  further and found that although the empirical Bayes method best identifies  future crashes, it is not as effective as the SafeStat model or the negative  binomial regression model in identifying carriers with the highest future  crash rates. The carriers identified with the empirical Bayes method were  invariably the largest carriers. This result is not especially useful from a  regulatory perspective. Companies operating a large number of vehicles  often have more crashes over a period of time than smaller companies.  However, this does not mean that the larger company is necessarily  violating more safety regulations or is less safe than the smaller company.  For this reason, we do not advocate the use of the empirical Bayes method  in conjunction with the negative binomial regression model as long as the  method used to calculate the safety evaluation area values remains  unchanged. If changes are made in how carriers are rated for safety, this  method may in the future offer more promise than the negative binomial  regression model alone.
section_title: Relationship of Carrier Financial Data and Safety Risk
paragraphs: Conducted on behalf of FMCSA, a study by Corsi, Barnard, and Gibney in  2002 examined how a carrier’s financial performance data correlate with  the carrier’s score on a compliance review. The authors selected those  motor carriers from MCMIS in December 2000 that had complete data for  the accident, driver, vehicle, and safety management safety evaluation  areas. Using these data, the authors then matched a total of 700 carriers to  company financial statements in the annual report database of the  American Trucking Associations. The authors created a binary response  variable for whether the carrier received a satisfactory or an  unsatisfactory outcome on the compliance review. The authors then  assessed how this result correlated with financial measures derived from  the company financial statements. In general, the study found that  indicators of poor financial condition correlated with an increased safety  risk.
Two practical considerations limit the applicability of the findings from  this study to SafeStat. First, the 700 carriers in the study sample are not  necessarily representative of the motor carriers that FMCSA oversees.  Only about 2 percent of the carriers evaluated by the SafeStat model in  June 2004 had a value for the safety management safety evaluation area.  Of these carriers, not all had complete data for the other three safety  evaluation areas. Second, FMCSA does not receive annual financial  statements from all motor carriers. For these reasons, we did not consider  using carrier financial data in our analysis of the SafeStat data.
section_title: Relationship of Commercial Driver License Convictions and Crash Risk
paragraphs: A series of studies by Lantz and others examined the effect of  incorporating conviction data from the state-run commercial driver license  data system into the calculation of a driver conviction measure. The  studies found that the driver conviction measure is weakly correlated with  the crash per vehicle rate. However, the studies did not incorporate the  proposed driver conviction measure into one of the existing safety  evaluation areas and use the updated measure to estimate new SafeStat  scores for carriers. While the use of commercial driver license conviction  data may have potential for future incorporation into a model for  identifying carriers that pose high crash risks, there is no assessment of its  impact at this time.
section_title: Impact of Data Quality on SafeStat’s Predictive Capability
paragraphs: The 2004 Office of Inspector General report, the 2004 Oak Ridge study, and  reports by the University of Michigan Transportation Research Institute on  state crash reporting all examined the impact of data quality on SafeStat’s  ability to identify carriers that pose high crash risks. These studies looked  at issues such as late reporting and incomplete or inaccurate reporting of  crash data and found weaknesses.
subsections: section_title: Late Reporting of Crash Data
paragraphs: To determine whether states promptly report SafeStat data, the Office of  Inspector General conducted a two-stage statistical sample in which it  selected 10 states for review and then selected crash and inspection  reports from those states for examination. It sampled 392 crash records  and 400 inspection records from July through December 2002. In 2 of the  10 states selected, Pennsylvania and New Mexico, no crash records were  available for the sample period, so it selected samples from earlier  periods. The Office of Inspector General also discussed reporting issues  with state and FMCSA officials and obtained crash records from selected  motor carriers. In addition, the Office of Inspector General used the  coefficient of variation to analyze data consistency and trends in reporting  timeliness across geographic regions. Our review of the study indicates  that it was based on sound audit methodology.
The study found that, as of November 2002, states submitted crash reports  in fiscal year 2002 an average of 103 days after the crash occurred and that  states varied widely in the timeliness of their crash data reporting.  (FMCSA requires that states report crashes no more than 90 days after  they occur.) In addition, the study found that 20 percent of the crashes  that occurred in fiscal year 2002 were entered into MCMIS 6 months or  more after the crash occurred. On the basis of this information, the Office  of Inspector General concluded that the calculation of the accident safety  evaluation area value was affected by the location of the carrier’s  operations but did not estimate the degree of this effect.
We also assessed the extent of late reporting. We measured how many  days, on average, it took each state to report crashes to MCMIS in each  calendar year and found that the amount of time taken to report crashes  declined from 2000 to 2005. Our findings were similar in nature to the  Office of Inspector General’s findings. However, our results are broader  because they are based on all crash data rather than a sample. In addition,  since our work is more recent, it reflects more current conditions. We  both came to the conclusion, although to varying degrees, that late  reporting of crash data by states negatively affects SafeStat’s identification  of carriers that pose high crash risks.
Oak Ridge also examined the impact of late reporting. Using data provided  by Volpe, Oak Ridge looked at the difference between the date a crash  occurred and the date it was entered into MCMIS. The researchers found  that after 497 days, 90 percent of the reported crashes were entered into  MCMIS.
The Oak Ridge study also reran the SafeStat model for March 2001 with  the addition of crash data from March 2003 to see how more complete  data changed SafeStat scores. The study found that the addition of late- reported data increased the number of carriers in the high-risk group by 18  percent. This late reporting affected the rankings of 8 percent of all the  carriers ranked by SafeStat in March 2001. Of these affected carriers, 3  percent moved to a lower SafeStat category, and 5 percent moved to a  higher category. Including the late-reported crash data available in March  2003 for the period from September 1998 through March 2001 resulted in a  35 percent increase in the available crash data.
We performed the same analysis as the Oak Ridge study and obtained  similar results. We used SafeStat data from June 2004, which include  carrier safety data from December 2001 through June 2004. Using  FMCSA’s master crash file from June 2006, we found that, with the  addition of late-reported crashes, 481 carriers moved into the highest risk  category, and 182 carriers dropped out of the highest risk category  resulting in a net increase of 299 carriers (6 percent) being added to the  highest risk category.
The University of Michigan Transportation Research Institute issued a  series of reports examining crash reporting rates in 14 states. These  reports looked at late reporting as a potential source of low crash  reporting rates but did not specifically examine the extent of late reporting  or the impact of late reporting on SafeStat scores. The institute looked at  reporting rates in each of the states by month to determine if reporting  rates were lower in the latter part of the year because of late reporting. It  found that reporting rates were lower in the latter part of the year in 6 of  the 14 states studied. This issue was not a focus of our efforts, so we did  not conduct a similar analysis.
section_title: Incomplete and Inaccurate Reporting of Crash Data
paragraphs: The Office of Inspector General’s study found several instances of  incomplete or inaccurate data on crashes and carriers. The study reviewed  MCMIS reporting for all states and found that 6 of them did not report any  crashes to FMCSA in the 6-month period from July through December  2002. In addition, the study found that MCMIS listed about 11 percent of  carriers as having no vehicles, and 15 percent as having no drivers. Finally,  from a sample of crash records, the study estimated that 13 percent of the  crash reports and 7 percent of the inspection reports in MCMIS contained  errors that would affect SafeStat results. In particular, the study concluded  that the database identified the wrong motor carrier as having been  involved in a crash or as having received a violation in 11 percent of the  erroneous records.
The University of Michigan Transportation Research Institute also  examined the accuracy of states’ crash data reporting. To determine if  crashes were reported accurately, the institute compared information  contained in the individual states’ police accident reporting files with  crash data reported to MCMIS. Some states, such as Ohio, had enough  information captured in the police accident file to determine if individual  crashes were eligible for reporting, and, therefore, the institute was able to  use these data in its analyses. In other states, not enough information was  available to make a determination, and the institute had to project results  on the basis of other states’ experience. The institute also carried out a  number of analyses, such as comparing reporting rates for different  reporting jurisdictions, in an attempt to identify reporting trends in the  individual states.
The institute identified several problems with the accuracy of states’ crash  reporting. All 14 states that it studied reported ineligible crashes to  MCMIS. These crashes were ineligible because they either involved  vehicles not eligible for reporting or they did not meet the crash severity  threshold. In total, the 14 states reported nearly 5,800 ineligible crashes to  MCMIS out of almost 68,000 crashes reported (9 percent). The states also  failed to report a number of eligible crashes: the 14 states studied reported  from 9 percent to 87 percent of eligible crashes.
Our review of the institute’s methodology indicates that its findings are  based on sound methodology and that its analyses were very thorough.  However, its studies are limited to the 14 states studied and to the  particular year studied. (Not all studies covered the same year.) These  states’ experience may or may not be representative of the experiences of  the entire country, and there is no way to determine if the reporting for  this year is representative of the state’s reporting activities over a number  of years or if the results were unique to that particular year. The  exceptions to this are the studies for Missouri, which covered calendar  years 2001 and 2005, and Ohio, which covered calendar years 2000 and  2005.
We did not attempt to assess the extent of inaccurate reporting in  individual states, but we did find examples of inaccurate data reporting.  To analyze the completeness of reporting, we attempted to match all crash  records in the MCMIS master crash file for crashes occurring between  December 26, 2001, and June 25, 2004, to the list of motor carriers in the  MCMIS census file. We found that Department of Transportation numbers  were missing for 30 percent of the crashes that were reported, and the  number did not match a Department of Transportation number listed in  MCMIS for 8 percent of reported crashes. We also compared the number  of crashes in MCMIS with the number in the General Estimates System  produced by the National Highway Traffic Safety Administration and  found evidence of underreporting of crashes to MCMIS.
section_title: Appendix II: Scope and Methodology
paragraphs: To determine whether statistical approaches could be used to improve  FMCSA’s ability to identify carriers that pose high crash risks, we tested a  variety of regression models and compared their results with results from  the existing SafeStat model. The models we tested, using MCMIS data used  by SafeStat in June 2004 to identify carriers that pose high crash risks,  include the Poisson, negative binomial, zero-inflated negative binomial,  zero-inflated Poisson, and empirical Bayes. We chose these regression  models because crash totals for a company represent count outcomes, and  these statistical models are appropriate for use with count data. In  addition, we explored logistic regression to assess the odds of having a  crash. Based on the results of the statistical models, we ranked the  predicted means (or predicted probabilities in the logistic regression) to  see which carriers would be at risk during the 18-month period after June  2004. We selected June 2004 because this date enabled us to examine  MCMIS data on actual crashes that occurred in the 18-month period from  July 2004 through December 2005. We used these data to determine the  degree to which SafeStat identified carriers that proved to pose high crash  risks. We then compared the predictive performance of the regression  models with the performance of SafeStat to determine which method best  identified carriers that pose high crash risks. Using a series of simple  random samples, we also calculated the crash rates of all carriers listed in  the main SafeStat summary results table in MCMIS for comparison with  the crash rates of carriers identified by SafeStat as high risk. We did this  analysis to determine whether the SafeStat model did a better job than  random selection of identifying motor carriers that pose high crash risks.
In addition, we tested changes to selected portions of the SafeStat model  to see whether improvements could be made in the identification of high- risk motor carriers. In one analysis, we modified the calculation of the  safety evaluation area values and compared the number of high-risk motor  carriers identified with the number identified by the unmodified safety  evaluation areas. For example, we included carriers with only one crash in  the calculation of the accident safety evaluation area whereas the  unmodified SafeStat model includes only carriers with two or more  crashes. We also investigated the effect of removing the time and severity  weights from the indexes used to construct the accident, driver, and  vehicle safety evaluation areas. We then compared the result of using the  modified and unmodified safety evaluation area values to determine if this  modification improved the model’s ability to identify future crash risks.
To assess the extent to which the timeliness, completeness, and accuracy  of MCMIS and state-reported crash data affect SafeStat’s performance, we  carried out a series of analyses with the MCMIS crash master file and  MCMIS census file, as well as surveying the literature to assess findings on  MCMIS data quality from other studies. To assess the effect of timeliness,  we first measured how many days on average it was taking each state to  report crashes to FMCSA by year for calendar years 2000 through 2005. We  also recalculated SafeStat scores from the model’s June 25, 2004, run to  include crashes that had occurred more than 90 days before that date but  had not been reported to FMCSA by that date. We compared the number  and rankings of carriers from the original SafeStat results with those  obtained by adding in data for the late-reported crashes. In addition, we  reviewed the University of Michigan Transportation Research Institute’s  studies of state crash reporting to MCMIS to identify the impact of late  reporting in individual states on MCMIS data quality.
To assess the effect of completeness, we attempted to match all crash  records in the MCMIS crash file for crashes occurring from December  2001 through June 2004 to the list of motor carriers in the MCMIS census  file. In addition, we reviewed the University of Michigan Transportation  Research Institute’s studies of state crash reporting to MCMIS to identify  the impact of incomplete crash reporting in individual states on MCMIS  data quality.
To assess the effect of accuracy, we reviewed a report by the Office of  Inspector General that tested the accuracy of electronic data by comparing  records selected in the sample with source paper documents. In addition,  we reviewed the University of Michigan Transportation Research  Institute’s studies of state crash reporting to MCMIS to identify the impact  of incorrectly reported crashes in individual states on MCMIS data quality.
While the limitations in the data adversely affect the ability of any method  to identify carriers that pose high crash risks, we determined that the data  were of sufficient quality for our use, which was to assess how the  application of regression models might improve the ability to identify high- risk carriers over the current approach—not to determine absolute  measures of crash risk. Our reasoning is based on the fact that we used the  same data set to compare the results of the SafeStat model and the  regression models. Limitations in the data would apply equally to both  results. Methods to identify carriers that pose high crash risk will perform  more efficiently once the known problems with the quality of state- reported crash data are addressed.
To understand what other researchers have found about how well SafeStat  identifies motor carriers that pose high crash risks, we identified studies  through a general literature review and by asking stakeholders and study  authors to identify high-quality studies. Studies included in our review  were (1) the 2004 study of SafeStat done by Oak Ridge National  Laboratory, (2) the SafeStat effectiveness studies done by the Department  of Transportation Office of Inspector General and Volpe Institute, (3) the  University of Michigan Transportation Research Institute’s studies of state  crash reporting to FMCSA, and (4) the 2006 Department of Transportation  Office of Inspector General’s audit of data for new entrant carriers. We  assessed the methodology used in each study and identified which  findings are supported by rigorous analysis. We accomplished this by  relying on information presented in the studies and, where possible, by  discussing the studies with the authors. When the studies’ methodologies  and analyses appeared reasonable, we used those findings in our analysis  of SafeStat. We discussed with FMCSA and industry and safety  stakeholders the SafeStat methodology issues and data quality issues  raised by these studies. We also discussed the aptness of the respective  methodological approaches with FMCSA. Finally, we reviewed FMCSA  documentation on how SafeStat is constructed and assessments of  SafeStat conducted by FMCSA.
section_title: Appendix III: Additional Results from Our Statistical Analyses of the SafeStat Model
paragraphs: This appendix contains technical descriptions and other  information related to our statistical analyses.
subsections: section_title: Overview of Regression Analyses
paragraphs: To study how well statistical methods identify carriers that pose high  crash risks, we carried out a series of regression analyses. The safety  evaluation area values for the accident, driver, vehicle, and safety  management areas served as the independent variables to predict crash  risks. We used the state-reported crash data in MCMIS for crashes that  occurred during the 30 months preceding June 25, 2004, as the dependent  variable in each model. We used the results of the SafeStat model run  from June 25, 2004, to benchmark the performance of the regression  models with the crash records for the identified high-risk carriers over  the succeeding 18 months.
We matched the state-reported crashes that occurred from December 26,  2001, through June 25, 2004, to the carriers listed in SafeStat. We checked  our match of crashes for carriers with those carriers used by FMCSA in  June 2004 and found that the reported numbers had changed for about  10,700 carriers in the intervening 2 years. We found this difference even  though we used only crashes that occurred from December 26, 2001,  through June 25, 2004, and were reported to FMCSA before June 25, 2004.  Because of this difference in matched crashes, we recalculated the  accident safety evaluation area using our match of the crashes. This is  discussed later in more detail.
Using our recalculation of the accident safety evaluation area values and  the original driver, vehicle, and safety management safety evaluation area  values for the carriers, we fit a Poisson regression model and a negative  binomial regression model to the crash counts. Both of these models are  statistically appropriate for use when modeling counts that are positive  and integer valued. The two models differ in their assumptions about the  mean and variance. Whereas the Poisson model assumes that the mean  and the variance are equal, the negative binomial model assumes the  mean is not equal to the variance. The crash data in MCMIS fit the  assumptions of the negative binomial distribution better than those of the  Poisson.
We also tried to estimate zero-inflated Poisson and zero-inflated negative  binomial models with the SafeStat data. These models are appropriate  when the count values include many zeros, as is the case with the values  in this data set (because many carriers do not have crash records).  However, we could not estimate the parameters for these models with the  MCMIS data. We also considered using logistic regression to model the  carrier’s odds of experiencing a crash. However, the parameter estimates  of the four safety evaluation area values could not be estimated, so we  did not use the results of this model.
Finally, we used the results from the negative binomial model to assess  the expected carrier crash counts using the empirical Bayes estimate. In  safety applications, the empirical Bayes method is used to increase the  precision of estimates and correct for the regression-to-mean bias. In this  application, the empirical Bayes method calculates a weighted average of  the rate of crashes for a carrier from the prior 30 months with the  predicted mean number of crashes from the negative binomial regression.  This method optimizes the identification of carriers with the highest  number of future crashes. This optimization of total crashes, however,  resulted in the identification of primarily the largest companies. The  crash rate (crashes per 1,000 vehicles per 18 months) was not as high for  this group as for the carriers placed by the SafeStat model in its A and B  categories.
section_title: Technical Explanation of the Negative Binomial Regression Model
paragraphs: This section provides the technical details for the negative binomial  regression model fit to the SafeStat data. This section also explains how  we handled incomplete safety evaluation area data for carriers in the  regression model analyses. ) ) ) exp( β ) )
This equation models the log of the expected mean number of crashes for  each motor carrier using the four safety evaluation area values, but most  commercial motor companies listed in MCMIS do not have values for all  four safety evaluation areas. To account for this, it is necessary to define  four indicator variables. Let  . β ) ) β ) )
A carrier has to have two or more reported crashes in the past 30 months to receive an  accident safety evaluation area value. A carrier has to have three or more roadside  inspections to receive a driver or vehicle safety evaluation area value. A driver has to have  had a compliance review in the past 18 months to receive a safety management safety  evaluation area value. There are other ways a carrier can receive a value for one of these  four safety evaluation areas, refer to the description of each one provided in the  Background. intercept for the indicator term (the coefficient for the indicator  function).
We used a similar parameterization to formulate the Poisson regression  model.
subsections: section_title: Evaluation of Regression Models’ Performance
paragraphs: We estimated regression models using the same data FMCSA used in its  application of the SafeStat model on June 25, 2004, with one exception  for the accident safety evaluation area. For that area, we used our own  match of crashes to carriers for December 26, 2001, through June 25,  2004. The MCMIS data we received in June 2006 produced different totals  in the match of crashes to carriers for about 10,700 carriers. MCMIS data  change over time because crash data are added, deleted, or changed as  more information about these crashes is obtained. The discrepancies in  matching arose even though we used the identical time interval and  counted crashes only when the record indicated they had been reported  to FMCSA before June 25, 2004. Because of these discrepancies, it was  necessary to calculate the accident safety evaluation area values using  our match of crashes and then recalculate the SafeStat carrier scores for  June 25, 2004, using our accident safety evaluation area values and the  original driver, vehicle, and safety management safety evaluation area  values. We used our accident safety evaluation area values and the  original driver, vehicle, and safety management safety evaluation area  values in the regression model analysis.
Using the revised accident safety evaluation area values and FMCSA’s  original driver, vehicle, and safety management safety evaluation area  values, the SafeStat model identified 4,989 carriers that pose high crash  risks. For each regression model, we input the safety evaluation area data  for the carriers in our analysis data set and used the regression model to  calculate the predicted mean number of crashes. We then sorted the  predicted scores and selected the 4,989 carriers with the worst predicted  values as the set of high-risk carriers identified by the regression model.  Next, we used MCMIS to determine the crash history of these 4,989  carriers between June 26, 2004, and December 25, 2005, and compared  the aggregate crash history with the aggregate crash history of the  carriers identified by the SafeStat model during the same period of time.
The regression models do not categorize carriers by letter; the regression  models produce a predicted crash risk for each carrier. The regression  models make use of the safety evaluation area values, but they differ from  the SafeStat model in this respect.
The results show that a negative binomial regression model estimated  with the safety evaluation area values outperforms the current SafeStat  model in terms of predicting future crashes and the future crash rate  among identified carriers that pose high crash risks. (See table 3.) That is,  our negative binomial and Poisson models show 111 and 109 crashes per  1,000 vehicles per 18 months, respectively, compared with the 102  crashes per 1,000 vehicles per 18 months estimated by the current  SafeStat model. The Poisson model is not as appropriate since the crash  counts for carriers have variability that is significantly different from the  mean number of crashes. The empirical Bayes method optimizes the  selection of future crashes; however, it does so by selecting the largest  carriers. The largest carriers have a lower crash rate per 1,000 vehicles  per 18 months than the carriers that pose high crash risks identified by  the SafeStat model or by the negative binomial regression model. Since  the primary use of SafeStat is to identify and prioritize carriers for  FMCSA and state compliance reviews, the empirical Bayes method did  not identify carriers with the highest safety risk.