Motor Carrier Safety: A Statistical Approach Will Better Identify Commercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach
Released: 2007-06-11T00:00:00

Letter

Background

The interstate commercial motor carrier industry, primarily the trucking industry, is an important part of the nation’s economy. Trucks transport over 11 billion tons of goods annually, or about 60 percent of the total domestic tonnage shipped. Buses also play an important role, transporting an estimated 631 million passengers annually. There are approximately 711,000 commercial motor carriers registered in MCMIS, about 9 million trucks and buses, and more than 10 million drivers. Most motor carriers are small; about 51 percent operate one vehicle, and another 31 percent operate two to four vehicles. Carrier operations vary widely in size, however, and some of the largest motor carriers operate upwards of 50,000 vehicles. Carriers continually enter and exit the industry. Since 1998, the industry has increased in size by an average of about 29,000 interstate carriers per year.

In the United States, commercial motor carriers account for less than 5 percent of all highway crashes, but these crashes result in about 13 percent of all highway deaths, or about 5,500 of the approximately 43,000 highway fatalities that occur nationwide annually. In addition, about 106,000 of the approximately 2.7 million highway injuries per year involve motor carriers. The fatality rate for trucks has generally been decreasing over the past 30 years, but this decrease has leveled off, and the rate has been fairly stable since the mid-1990s. The fatality rate for buses has improved slightly from 1975 to 2005 but has more annual variability than the fatality rate for trucks due to a much smaller total vehicle miles traveled. (See fig. 1.)

Congress created FMCSA through the Motor Carrier Safety Improvement Act of 1999 to reduce crashes, injuries, and fatalities involving commercial motor vehicles. To accomplish this mission, FMCSA carries out a number of enforcement, education, and outreach activities. FMCSA uses enforcement as its primary approach for reducing the number of crashes, fatalities, and injuries involving trucks and buses. Some of FMCSA’s enforcement programs include compliance reviews, which are on-site reviews of carriers’ records and operations to determine compliance with regulations; safety audits of new interstate carriers; and roadside inspections of drivers and vehicles.

FMCSA’s education and outreach programs are intended to promote motor carrier safety and consumer awareness. One of the programs is the New Entrant program, which is designed to inform newly registered motor carriers about motor carrier safety standards and regulations to help them comply with FMCSA’s requirements. Other programs are designed to identify unregistered carriers and get them to register, promote increased safety belt use among commercial drivers, and inform organizations and individuals that hire buses how to make safe choices. FMCSA plans to make major revisions to its compliance and enforcement approach under an initiative called Comprehensive Safety Analysis 2010.

Compliance reviews are an important enforcement tool because they allow FMCSA to take an in-depth look at carriers that have been identified as posing high crash risks because of high crash rates or poor safety performance records. Motor carriers may be identified as high risk from SafeStat or through calls to FMCSA’s complaint hotline. Carriers are given a satisfactory, conditional, or unsatisfactory safety rating. A conditional rating means the carrier is allowed to continue operating, but FMCSA may schedule a follow-up compliance review to ensure that problems noted in the first compliance review are addressed. An unsatisfactory rating must be addressed or the carrier is placed out of service, meaning it is no longer allowed to do business, and the carrier may face legal enforcement actions undertaken by FMCSA. Compliance reviews can take several days to complete, depending on the size of the carrier, and may result in enforcement actions being taken against a carrier.

FMCSA uses both its own inspectors and state inspectors to carry out its enforcement activities. In total, about 750 staff are available to perform compliance reviews, and more than 10,000 staff do vehicle and driver inspections at weigh stations and other points. Together, FMCSA and its state partners perform about 16,000 compliance reviews a year, which cover about 2 percent of the nation’s 711,000 carriers.

Because the number of inspectors is small compared with the size of the motor carrier industry, FMCSA prioritizes carriers for compliance reviews. To do so, it uses SafeStat to identify carriers that pose high crash risks. SafeStat is a model that uses information gathered from crashes, roadside inspections, traffic violations, compliance reviews, and enforcement cases to determine a motor carrier’s safety performance relative to that of other motor carriers that have similar exposure in these areas. A carrier’s score is calculated on the basis of its performance in four safety evaluation areas: Accident safety evaluation area: The accident safety evaluation area reflects a carrier’s crash history relative to other motor carriers’ histories. The safety evaluation area is based on state-reported crash data, vehicle data from MCMIS, and data on reportable crashes and annual vehicle miles traveled from the most recent compliance review. A carrier must have two or more reportable crashes within the last 30 months to have the potential to receive a deficient value and thus be made a priority for a compliance review.

Driver safety evaluation area: The driver safety evaluation area reflects a carrier’s driver-related safety performance and compliance relative to other motor carriers. The driver safety evaluation area is based on violations cited in roadside inspections that have been performed within the last 30 months and compliance reviews that have occurred within the last 18 months, together with the number of drivers listed in MCMIS. A carrier must have three or more driver inspections, three or more moving violations, or at least one acute or critical violation of driver regulations from a compliance review to have the potential to receive a deficient value and thus be made a priority for a compliance review.

Vehicle safety evaluation area: The vehicle safety evaluation area reflects a carrier’s vehicle-related safety performance and compliance relative to other motor carriers. The vehicle safety evaluation area is based on violations identified during vehicle roadside inspections that have occurred within the last 30 months or vehicle-related acute and critical violations of regulations discovered during compliance reviews that have occurred within the last 18 months. A carrier must have either three or more vehicle inspections or at least one acute or critical violation of vehicle regulations from a compliance review to have the potential to receive a deficient value and thus be made a priority for a compliance review.

Safety management safety evaluation area: The safety management safety evaluation area reflects a carrier’s safety management relative to other motor carriers. It is based on the results of violations cited in closed enforcement cases in the past 6 years or violations of regulations related to hazardous materials and safety management discovered during a compliance review performed within the last 18 months. A carrier must have had at least one enforcement case initiated and closed or at least two enforcement cases closed within the past 6 years, or at least one acute, critical, or severe violation of hazardous material or safety management regulations identified during a compliance review within the last 18 months to have the potential to receive a deficient value and thus be made a priority for a compliance review.

A motor carrier’s score is based on its relative ranking, indicated as a value, in each of the four safety evaluation areas. For example, if a carrier receives a value of 75 in the accident safety evaluation area, then 75 percent of all carriers with sufficient data for evaluation performed better in that safety evaluation area, while 25 percent performed worse. The calculation used to determine a motor carrier’s SafeStat score is as follows: SafeStat Score = (2.0x accident value) + (1.5x driver value) + vehicle value + safety management value As shown in the formula, the accident and driver safety evaluation areas have 2.0 and 1.5 times the weight, respectively, of the vehicle and safety management safety evaluation areas. Safety evaluation area values less than 75 are ignored in the formula used to determine the SafeStat score. For example, a carrier with values of 74 for all four safety evaluation areas has a SafeStat score of 0. FMCSA assigned more weight to these safety evaluation areas because, according to FMCSA, crashes and driver violations correlate relatively better with future crash risk. In addition, more weight is assigned to fatal crashes and to crashes that occurred within the last 18 months. In consultation with state transportation officials, insurance industry representatives, safety advocates, and the motor carrier industry, FMCSA used its expert judgment and professional knowledge to assign these weights, rather than determining them through a statistical approach, such as regression modeling.

FMCSA assigns carriers categories ranging from A to H according to their performance in each of the safety evaluation areas. A carrier is considered to be deficient in a safety evaluation area if it receives a value of 75 or higher in that particular safety evaluation area. Although a carrier may receive a value in any of the four safety evaluation areas, the carrier receives a SafeStat score only if it is deficient in one or more safety evaluation areas. Carriers that are deficient in two or more safety evaluation areas and have a SafeStat score of 225 or more are considered to pose high crash risks and are placed in category A or B. (See table 1.) Carriers that are deficient in two safety evaluation areas but have a SafeStat score of less than 225 are placed in category C and receive a medium priority for compliance reviews. Carriers that are deficient in only one of the safety evaluation areas are placed in category D, E, F, or G. Carriers that are not deficient in any of the safety evaluation areas do not receive a SafeStat score and are placed in category H.

Of the 622,000 motor carriers listed in MCMIS as having one or more vehicles in June 2004, about 140,000, or 23 percent, received a SafeStat category A through H. There are several reasons why a small proportion of carriers receive a score. First, approximately 305,900, or about 42 percent, of the carriers have crash, vehicle inspection, driver inspection, or enforcement data of any kind. SafeStat relies on these data to calculate a motor carrier’s score, so carriers without such data are not rated by SafeStat. It is likely that some of the carriers listed in MCMIS are no longer in business, but it is also possible that these carriers had no crashes, inspections, or compliance reviews in the 30-month period prior to June 2004. Second, a carrier must meet the minimum requirements to be assigned a value in a given safety evaluation area. If, for example, a carrier had only one reportable crash within the last 30 months, then the carrier would not be assigned an accident safety evaluation area value. Of the 305,900 carriers that have any safety data in SafeStat, 140,000 met the SafeStat minimum requirements in one or more safety evaluation areas. Of these 140,000 carriers, 45,000 were rated in categories A through G. The other carriers were placed in category H because they were not considered deficient, meaning they did not receive a value of 75 or more in any of the safety evaluation areas.

The design of SafeStat and its data sufficiency requirements increase the likelihood that larger motor carriers will be deficient in one of the safety evaluation areas, in other words, rated in categories A through G, than are small carriers. About 51 percent of all carriers listed in MCMIS operate one vehicle, and about 3 percent of them received a SafeStat rating in categories A through G. (See table 2.) In contrast, fewer than 1 percent of the carriers listed in MCMIS have more than 100 vehicles, and nearly 25 percent of them received a SafeStat rating in categories A through G.

A Statistical Approach Would Better Identify Carriers That Pose High Crash Risks Than Does FMCSA’s Current Approach

We found that FMCSA could improve SafeStat’s ability to identify carriers that pose high crash risks if it applied a statistical approach, called a negative binomial regression model, to the four SafeStat safety evaluation areas instead of its current approach. Through this change, FMCSA could more efficiently target compliance reviews to the set of carriers that pose the greatest crash risk. Applying a negative binomial regression model would improve the identification of high risk carriers over SafeStat’s performance by about 9 percent, compared with the current approach, which incorporates safety data weighted in accordance with the professional judgment and experience of SafeStat’s designers. Moreover, according to our analysis, this 9 percent improvement would enable FMCSA to identify carriers with almost twice as many crashes in the following 18 months as those carriers identified under its current approach. Targeting these high-risk carriers would result in FMCSA giving compliance reviews to carriers that experienced both a higher crash rate and, in conjunction with the higher crash rate, 9,500 more crashes over an 18-month period than those identified by the SafeStat model. Applying a negative binomial regression model approach to the SafeStat safety evaluation areas would be easy to implement and, in our opinion, would be consistent with other FMCSA uses for SafeStat beyond identifying carriers that pose high risks for crashes. In addition, adopting a negative binomial regression model approach would be beneficial even if FMCSA makes major revisions to its compliance and enforcement approach in the coming years under its Comprehensive Safety Analysis 2010 initiative. Overall, other changes to the SafeStat model that we explored, such as modifying decision rules used in the construction of the safety evaluation areas, did not improve the model’s overall performance.

Regression Models Identify Carriers That Pose High Crash Risks Better Than Expert Judgment

Although SafeStat is nearly twice as effective as (83 percent better than) random selection in identifying carriers that pose high crash risks and, therefore, has value for improving safety, we found that FMCSA could improve SafeStat’s ability to identify such carriers by about 9 percent if it applied a negative binomial regression model approach to its analysis of motor carrier safety data. The use of a regression model does not entail assigning the letter categories currently assigned by the SafeStat model. Rather, the model predicts carriers’ crash risks, sorts the carriers according to their risk level, and assigns a high priority for a compliance review to the highest risk carriers. The improvement in identification of high-risk carriers, which we observed with the negative binomial regression model, is consistent with results obtained in an earlier analysis of MCMIS data performed by a team of researchers at Oak Ridge National Laboratory.

To compare the effectiveness of regression models and SafeStat in identifying carriers that pose high crash risks, we applied several regression models to the four safety evaluation areas (accident, driver, vehicle, and safety management) used by the SafeStat model. We recalculated SafeStat’s June 2004 accident safety evaluation area values because the data FMCSA provided on the number of crashes for each carrier differed in 2006 from the data used in the model in 2004. Using our accident safety evaluation area value and the original driver, vehicle, and safety management safety evaluation area values from June 2004, we selected the 4,989 carriers that our regression models identified as the highest crash risks, calculated the crash rate per 1,000 vehicles for these carriers over the next 18 months, and compared this rate with the crash rate per 1,000 vehicles for the 4,989 carriers identified by the SafeStat model as posing high crash risks (categories A and B).

All of the regression models that we estimated were at least as effective as SafeStat in identifying motor carriers that posed high crash risks. (See app. III for these results.) Of these, the negative binomial regression approach gave the best results and proved 9 percent more effective than SafeStat, as measured by future crashes per 1,000 vehicles. The set of carriers in SafeStat categories A and B had a crash rate of 102 per 1,000 vehicles for the 18 months after June 2004 while the set of high-risk carriers identified by the negative binomial regression model had 111 crashes per 1,000 vehicles. Even though this 9 percent improvement rate seems modest, it translates into nearly twice as many “future crashes” identified. Specifically, the negative binomial regression model identified carriers that had nearly twice as many crashes (from July 2004 to December 2005) as the carriers identified by SafeStat—19,580 crashes compared with 10,076.

SafeStat (categories A and B) and our negative binomial regression model identified many of the same carriers—1,924 of the 4,989 (39 percent)—as posing high crash risks. However, our model also identified a number of high-risk carriers that SafeStat did not identify, and vice versa. For example, our model identified 2,244 carriers as posing high crash risks, while SafeStat placed these carriers in category D (the accident area), assigning them a lower priority for compliance reviews. One reason for this difference is the decision rules that SafeStat employs. Under SafeStat, carriers must perform worse than 75 percent of all carriers to be considered deficient in any safety evaluation area. The regression approach identifies the carriers with the highest crash risks regardless of how they compare with their peers in individual areas. For example, we identified as posing high crash risks 482 carriers that SafeStat did not consider at all for compliance reviews because the carriers had not performed worse than 75 percent of their peers in any of the four safety evaluation areas.

FMCSA Can Apply a Regression Model Approach in the Short Term, Even Though It Is Planning to Overhaul SafeStat

In the short term, FMCSA could easily implement a regression model approach for SafeStat. All the information required as input for the negative binomial regression model is already entered into SafeStat. In addition, a standard statistical package can be used to apply the negative binomial approach to the four SafeStat safety evaluation areas. Like SafeStat, the negative binomial regression model would be run every month to produce a list of motor carriers that pose high crash risks, and these carriers would then be assigned priorities for a compliance review. As with SafeStat, the results of the negative binomial model would change slightly each month with the addition of new safety data to MCMIS.

In discussing the concept of adopting a negative binomial regression model approach with FMCSA officials, they were interested in understanding how the use of the negative binomial regression model results could be used to identify and improve the safety of those carriers that pose the greatest crash risks (much as the SafeStat categories of A and B do now) and how it could employ the proposed approach for current uses beyond identifying carriers that pose high crash risks. These uses include providing an understandable public display to shippers, insurers, and others who are interested in the safety of carriers; selecting carriers for roadside inspections; and trying to gain carriers’ compliance with driver and vehicle safety rules, when these carriers may not have crashes, consistent with agency efforts.

Identifying and improving the safety of carriers that pose high crash risks. The negative binomial regression model approach would produce a rank order listing of carriers by crash risk and by the predicted number of crashes. For compliance reviews, FMCSA could choose those carriers with the greatest number of predicted crashes. FMCSA would choose the number of carriers to review based on the resources available to it, much as it currently does.

Regarding improving the safety of carriers that pose high crash risks, FMCSA currently enrolls carriers that receive a SafeStat category of A, B, or C in the Motor Carrier Safety Improvement Program. This program aims to improve the safety of high-risk carriers through (1) a repetitive cycle of identification, data gathering, and assessment and (2) progressively harsher treatments applied to carriers that do not improve their safety. The use of a negative binomial regression model would not affect the structure or workings of this program, other than to better identify carriers that pose high crash risks. As discussed above, FMCSA would use the regression model’s results to identify the highest risk carriers and then intervene using its existing approaches (such as issuing warning letters, conducting follow-up compliance reviews, or levying civil penalties) as treatment.

Providing an understandable display to the public. FMCSA could choose to provide a rank order listing of carriers together with the associated number of predicted crashes or it could look for natural breaks in the predicted number of crashes and associate a category—such as “category A” to these carriers.

Selecting carriers for roadside inspections. Safety rankings from the SafeStat model are also used in FMCSA’s Inspection Selection System to prioritize carriers for roadside driver and vehicle inspections. The negative binomial regression model optimizes the identification of carriers by crash risk using safety evaluation area information. The negative binomial regression model approach that we describe in this report retains SafeStat’s basic design with four safety management areas (driver, vehicle, accident, and safety management). Therefore, FMCSA could use the negative binomial regression model results to identify carriers that pose a high crash risk, the results from the driver and vehicle safety evaluation areas, or both, to target carriers or vehicles for roadside driver and vehicle inspections.

Furthering agency efforts to gain compliance with driver and vehicle safety rules for carriers that do not experience crashes (or a sufficient number of crashes to pose a high risk for crashes). FMCSA was interested in understanding how, if at all, the negative binomial regression model approach would affect gaining compliance against carriers that may routinely violate safety rules (such as drivers’ hours of service requirements), but where these violations do not lead to crashes. As discussed above, the negative binomial regression model approach retains SafeStat’s four safety evaluation areas. Where it differs, is that it assigns different weights to those areas based on a statistical procedure, rather than having the weights assigned by expert judgment. As a result, FMCSA would still be able to identify carriers with many driver, vehicle, and safety management violations.

Other opportunities also exist for FMCSA to improve the ability of regression models to identify carriers that pose high crash risks. In 2005, a FMCSA compliance review work group reported a positive correlation between driver hours of service violations and crash rates. Because FMCSA can link violations of specific regulatory provisions, including those limiting driver hours of service, to the crash experience of the carriers involved, it has the opportunity to improve the violation severity weighting used in constructing the driver and vehicle safety evaluation areas. FMCSA has detailed violation data from roadside inspections and can statistically analyze these data to find other strong relationships with carriers’ crash risks. Changes made to the safety evaluation area methodology to strengthen the association with crash risk will improve the ability of the negative binomial regression model to identify carriers that pose high crash risks.

FMCSA has expressed doubts in the past when analysts have proposed switching to a regression model approach. For example, Oak Ridge National Laboratory advocated using a regression model approach in place of SafeStat in 2004, but FMCSA was reluctant to move away from its expert judgment model because it believed that the regression model approach would place undue weight on the accident safety evaluation area in determining priorities for compliance reviews, thereby diminishing the incentive for motor carriers to comply with the many safety regulations that feed into the driver, vehicle, and safety management safety evaluation areas. In FMCSA’s view, carriers would be less likely to comply with these regulations because violations in the driver, vehicle, and safety management areas would be less likely to lead to compliance reviews under a regression model approach that placed a heavy emphasis on crashes. Our view is that adopting a negative binomial regression model approach would better identify carriers that pose high crash risks and would thus further FMCSA’s primary mission of ensuring safe operating practices among commercial interstate motor carriers.

Over the longer term, FMCSA is considering a complete overhaul of its safety fitness determinations with its Comprehensive Safety Analysis 2010 initiative. This planned comprehensive review and analysis of the agency’s compliance and enforcement programs may result in a new operational model for identifying drivers and carriers that pose safety problems and for intervening to address those problems. FMCSA expects to deploy the results of this initiative in 2010. In our opinion, given the relative ease of adopting the regression modeling approach discussed in this report, and the immediate benefits that can be achieved, there is no reason to wait for FMCSA to complete its initiative, even if the initiative results in major revisions to the SafeStat model.

Modifications of SafeStat Did Not Improve Crash Identification

Besides investigating whether the use of regression models could improve SafeStat’s ability to identify carriers that pose high crash risks, we explored whether the existing model could be improved by changing several of its decision rules. Overall, these changes did not enhance the model’s ability to identify carriers that pose high crash risks. As long as FMCSA continues to estimate the safety evaluation area values with its present methodology, the rules we investigated help make the identification of high-risk motor carriers more efficient for both SafeStat and the negative binomial regression model.

Because the SafeStat model is composed of many components, we selected three decision rules for analysis. We chose these three rules because they are important pillars of the SafeStat model’s methodology for constructing the safety evaluation areas and because we could complete our analysis of them during the time we had to perform our work. A fuller exploration of areas with high potential to improve the identification of carriers that pose high crash risks would be a long-term effort, and FMCSA plans to address this work as part of the Comprehensive Safety Analysis 2010 initiative.

Removing comparison groups. As part of its methodology for calculating the accident, driver, and vehicle safety evaluation area values, SafeStat divides carriers into comparison groups. For example, in the driver safety evaluation area, SafeStat groups carriers by the number of moving violations they have, placing them in one of four groups (3 to 9, 10 to 28, 29 to 94, and 95 or more). SafeStat uses the comparison groups to control for the size of the carrier. We removed all the comparison groups in each of the three safety evaluation areas, recalculated their values, and compared the number of crashes in which the carriers were involved and their crash rates, for each of the SafeStat categories A through H, with the SafeStat results in which comparison groups were retained.

Removing minimum event requirements. SafeStat imposes minimum event requirements. For example, as noted, SafeStat does not consider a carrier’s moving violations if, in the aggregate, its drivers had fewer than three moving violations over a 30-month period. FMCSA does not calculate a safety evaluation area value for carriers with fewer than three events in an attempt to control for carriers that have infrequent, rather than possibly systemic, safety problems. We removed the requirement to have a minimum number of events (such as moving violations, crashes, and inspections), recalculated the three safety evaluation values, and compared the number of crashes in which the carriers were involved and their crash rates, for each of the SafeStat categories A through H, with the SafeStat results in which minimum event requirements were retained.

Removing time and severity weights. The SafeStat formula weights more recent events and more severe events more heavily than less recent or less severe events in the accident, driver, and vehicle safety evaluation areas. For example, the results of vehicle roadside inspections performed within the latest 6 months receive three times the weight of inspections performed 2 years ago. Similarly, crashes involving deaths or injuries receive twice as much weight as those that resulted in property damage only. We removed the time and severity weights for the three safety evaluation areas, recalculated these values, and compared the number of crashes in which the carriers were involved and their crash rates, for each of the SafeStat categories A through H, with the SafeStat results in which time and severity weights were retained.

Simultaneous changes to comparison group, event, and time severity requirements. Finally, we simultaneously removed comparison groups, minimum event requirements, and time and severity weights and compared the number of crashes in which the carriers were involved and their crash rates, for each of the SafeStat categories A through H, with the SafeStat results in which comparison groups, minimum event requirements, and time and severity weights were retained.

The results of each of our individual analyses and of making all changes simultaneously produced one of two outcomes, neither of which was considered more desirable. Relaxing the minimum data requirements greatly increased the number of carriers identified as high risk without increasing the overall number of predicted crashes over the subsequent 18 months, thus reducing the effectiveness of the SafeStat model. Removing comparison groups and removing time and severity weights had the effect of reducing the future crashes per 1,000 vehicles among those carriers identified as high risk, also reducing the effectiveness of the SafeStat model. As a result, we are not reporting on these results in detail. Trying to modify the decision rules used in SafeStat did highlight the balance that FMCSA has to strike between maximizing the identification of companies with the largest number of crashes (usually larger carriers) and those carriers with the greatest safety risk (which can be of any size).

Despite Quality Problems, FMCSA’s Crash Data Can Be Used to Compare Methods for Identifying Carriers That Pose High Crash Risks

The quality of crash data is a long-standing problem that potentially hindered FMCSA’s ability to accurately identify carriers that pose high crash risks. Despite the problems of late-reported crashes and incomplete and inaccurate data on crashes during the period we studied, we determined that the data were of sufficient quality for our use, which was to assess how the application of regression models might improve the ability to identify high-risk carriers over the current approach—not to determine absolute measures of crash risk. Our reasoning is based on the fact that we used the same data set to compare the results of the SafeStat model and the regression models. Limitations in the data would apply equally to both results. FMCSA has recently undertaken a number of efforts to improve crash data quality.

Late Reporting Had a Small Effect on SafeStat’s Ability to Identify High- Risk Carriers

FMCSA’s guidance provides that states report all crashes to MCMIS within 90 days of their occurrence. Late reporting can cause SafeStat to miss some of the carriers that should have received a SafeStat score. Alternatively, since SafeStat’s scoring involves a relative ranking of carriers, a carrier may receive a SafeStat score and have to undergo a compliance review because crash data for a higher risk carrier were reported late and not included in the calculation.

Late reporting affected SafeStat’s ability to identify all high-risk carriers to a small degree—about 6 percent-—for the period that we studied. Late reporting of crashes by states affected the safety rankings of more than 600 carriers, both positively and negatively. When SafeStat analyzed the 2004 data, which did not include the late-reported crashes, it identified 4,989 motor carriers as highest risk, meaning they received a category A or B ranking. With the addition of late-reported crashes, 481 carriers moved into the highest risk category, and 182 carriers dropped out of the highest risk category, resulting in a net increase of 299 carriers (6 percent) in the highest risk category. After the late-reported crashes were added, 481 carriers that originally received a category C, D, E, F, or G SafeStat rating received an A or B rating. These carriers would not originally have been given a high priority for a compliance review because the SafeStat calculation did not take into account all of their crashes. On the other hand, a small number of carriers would have received a lower priority if the late-reported crashes had been included in their score. Specifically, 182 carriers – or fewer than 4 percent of those ranked, fell from the A or B category into the C, D, E, F, or G category once the late-reported crashes were included. These carriers would not have been considered high priority for a compliance review if all crashes had been reported on time. This does not have a big effect on the overall motor carrier population, however, as only 4 percent of carriers originally identified as highest risk were negatively affected by late reporting.

The timeliness of crash reporting has shown steady and marked improvement. The median number of days it took states to report crashes to MCMIS dropped from 225 days in calendar year 2001 to 57 days in 2005 (the latest data available at the time of our analysis). In addition, the percentage of crashes reported by states within 90 days of occurrence has jumped from 32 percent in fiscal year 2000 to 89 percent in fiscal year 2006. (See fig. 2.)

Incomplete Data from States Potentially Limit SafeStat’s Identification of All Carriers That Pose High Crash Risks

FMCSA uses a motor carrier identification number, which is unique to each carrier, as the primary means of linking inspections, crashes, and compliance reviews to motor carriers. Approximately 184,000 (76 percent) of the 244,000 crashes reported to MCMIS between December 2001 and June 2004 involved interstate carriers. Of these 184,000 crashes, nearly 24,000 (13 percent) were missing this identification number. As a result, FMCSA could not match these crashes to motor carriers or use them in SafeStat. In addition, the carrier identification number could not be matched to a number listed in MCMIS for 15,000 (8 percent) other crashes that involved interstate carriers. Missing data or being unable to match data for nearly one quarter of the crashes during the period of our review potentially has a large impact on a motor carrier’s SafeStat score because SafeStat treats crashes as the most important source of information for assessing motor carrier crash risk. Theoretically, information exists to match crash records to motor carriers by other means, but such matching would require too much manual work to be practicable.

We were not able to quantify the actual effect of either the missing data or the data that could not be matched for MCMIS overall. To do so would have required us to gather crash records at the state level—an effort that was impractical. For the same reason, we cannot quantify the effects of FMCSA’s efforts to improve the completeness of the data (discussed later). However, a series of reports by the University of Michigan Transportation Research Institute sheds some light on the completeness of the data submitted to MCMIS by the states. One of the goals of the research was to determine the states’ crash reporting rates. Reporting rates varied greatly among the 14 states studied, ranging from 9 percent in New Mexico in 2003 to 87 percent in Nebraska in 2005. It is not possible to draw wide-scale conclusions about whether state reporting rates are improving over time because only two of the states—Missouri and Ohio—- were studied in multiple years. However, in these two states, the reporting rate did improve. Missouri experienced a large improvement in its reporting rate, with 61 percent of eligible crashes reported in 2001, and 83 percent reported in 2005. Ohio’s improvement was more modest, increasing from 39 percent in 2000 to 43 percent in 2005.

The University of Michigan Transportation Research Institute’s reports also identified a number of factors that may affect states’ reporting rates. One of the main factors affecting reporting rates is the reporting officer’s understanding of crash reporting requirements. The studies note that reporting rates are generally lower for less serious crashes and for crashes involving smaller vehicles, which may indicate that there is some confusion about which crashes are reportable. Some states, such as Missouri, aid the officer by explicitly listing reporting criteria on the police accident reporting form, while other states, such as Washington, leave it up to the officer to complete certain sections of the form if the crash is reportable, but the form includes no guidance on reportable crashes. Yet other states, such as North Carolina and Illinois, have taken this task out of officers’ hands and include all reporting elements on the police accident reporting form. Reportable crashes are then selected centrally by the state, and the required data are transmitted to MCMIS.

Inaccurate Data Potentially Limit SafeStat’s Ability to Identify Carriers That Pose High Crash Risks

Inaccurate data, such as reporting a nonqualifying crash to FMCSA, potentially has a large impact on a motor carrier’s SafeStat score because SafeStat treats crashes as the most important source of information for assessing motor carrier crash risk. For the same reasons as discussed in the preceding section, we were neither able to quantify these effects nor determine how data accuracy has improved for MCMIS overall.

The University of Michigan Transportation Research Institute’s reports on crash reporting show that, among the 14 states studied, incorrect reporting of crash data is widespread. In recent reports, the researchers found that, in 2005, Ohio incorrectly reported 1,094 (22 percent) of the 5,037 cases, and Louisiana incorrectly reported 137 (5 percent) of the 2,699 cases. In Ohio, most of the incorrectly reported crashes did not qualify because they did not meet the crash severity threshold. In contrast, most of the incorrectly reported crashes in Louisiana did not qualify because they did not involve vehicles eligible for reporting. Other states studied by the institute had similar problems with reporting crashes that did not meet the criteria for reporting to MCMIS. These additional crashes could cause some carriers to exceed the minimum number of crashes required to receive a SafeStat rating and result in SafeStat’s mistakenly identifying carriers as posing high crash risks. Because each report focuses on reporting in one state in a particular year, it is not possible to identify the number of cases that have been incorrectly reported nationwide and, therefore, it is not possible to determine the impact of inaccurate reporting on SafeStat’s calculations.

As noted in the University of Michigan Transportation Research Institute’s reports, states may be unintentionally submitting incorrect data to MCMIS because of difficulties in determining whether a crash meets the reporting criteria. For example, in Missouri, pickups are systematically excluded from MCMIS crash reporting, which may cause the state to miss reportable crashes. However, some pickups may have vehicle weights above the reporting threshold, making crashes involving them eligible for reporting. There is no way for the state to determine which crashes involving pickups qualify for reporting without examining the characteristics of each vehicle. In this case, the number of omissions is likely to be relatively small, but this example demonstrates the difficulty states may face when identifying reportable crashes.

In addition, in some states, the information contained in the police accident report may not be sufficient for the state to determine if a crash meets the accident severity threshold. It is generally straightforward to determine whether a fatality occurred as a result of a crash, but it may be difficult to determine whether an injured person was transported for medical attention or a vehicle was towed because of disabling damage. In some states, such as Illinois and New Jersey, an officer can indicate on the form if a vehicle was towed by checking a box, but there is no way to identify whether the reason for towing was disabling damage. It is likely that such uncertainty results in overreporting because some vehicles may be towed for other reasons.

FMCSA Has Undertaken Efforts to Improve Crash Data Quality

FMCSA has taken steps to try and improve the quality of crash data reporting. As we noted in November 2005, FMCSA has undertaken two major efforts to help states improve the quality of crash data. One program, the Safety Data Improvement Program, has provided funding to states to implement or expand activities designed to improve the completeness, timeliness, accuracy, and consistency of their data. FMCSA has also used a data quality rating system to rate and display ratings for states’ crash and inspection data quality. Due to its public nature, this map serves as an incentive for states to make improvements in their data quality.

To further improve these programs, FMCSA has made additional grants available to states and implemented our recommendations to (1) establish specific guidelines for assessing states’ requests for funding to support data improvement in order to better assess and prioritize the requests and (2) increase the usefulness of its state data quality map as a tool for monitoring and measuring commercial motor vehicle crash data by ensuring that the map adequately reflects the condition of the states’ commercial motor vehicle crash data.

In February 2004, FMCSA implemented Data Q’s, an online system that allows for challenging and correcting erroneous crash or inspection data. Users of this system include motor carriers, the general public, state officials, and FMCSA. In addition, in response to a recent recommendation by the Department of Transportation Inspector General, FMCSA is planning to conduct a number of evaluations of the effectiveness of a training course on crash data collection that it will be providing to states by September 2008.

While the quality of crash reporting is sufficient for use in identifying motor carriers that pose high crash risks and has started to improve, commercial motor vehicle crash data continue to have some problems with timeliness, completeness, and accuracy. These problems have been well-documented in several studies, and FMCSA is taking steps to address the problems through studies of each state’s crash reporting system and grants to states to fund improvements. As a result, we are not making any recommendations in this area.

Conclusion

Interstate commerce involving large trucks and buses has been growing substantially, and this growth is expected to continue. While the number of fatalities per million vehicle miles traveled has generally decreased over the last 30 years, the fatality rate has leveled off and remained fairly steady since the mid-1990s. FMCSA could more effectively address fatalities due to crashes involving a commercial motor vehicle if it better targeted compliance reviews to those carriers that pose the greatest crash risks. Using a negative binomial regression model would further FMCSA’s mission of reducing crashes through the more effective targeting of compliance reviews to the set of carriers that pose the greatest crash risks. In light of possible changes to FMCSA’s safety fitness determinations resulting from its Comprehensive Safety Analysis 2010 initiative, we are not suggesting that FMCSA undertake a complete and thorough investigation of SafeStat. Rather, we are advocating that FMCSA apply a statistical approach that employs the negative binomial regression model rather than relying on the current SafeStat formula that was determined through expert judgment. In our view, the substitution of a statistically based approach would likely yield a markedly better ability to identify carriers that pose high crash risks with relatively little time or effort on FMCSA’s part.

Recommendation for Executive Action

We recommend that the Secretary of Transportation direct the Administrator of FMCSA to apply a negative binomial regression model, such as the one discussed in this report, to enhance the current SafeStat methodology.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Transportation for its review and comment. In response, departmental officials, including FMCSA’s Director of the Office of Enforcement and Compliance and Director of the Office of Research and Analysis, noted that our report provided useful insights and offered a potential avenue for further improving the effectiveness of FMCSA’s efforts to reduce crashes involving motor carriers. The agency indicated that it is already working to improve upon SafeStat as part of its Comprehensive Safety Analysis 2010 initiative. FMCSA agreed that it would be useful for it to consider whether there are both short and longer term measures that would incorporate the type of analysis identified in our report, as an adjunct to the SafeStat model, in order to better target compliance reviews so as to make the best use of FMCSA’s resources to reduce crashes.

The agency expressed some concerns with the negative binomial regression analysis, noting that its intent is to effectively target its compliance activities based on a broader range of factors than is considered in the negative binomial regression analysis approach described in our draft report, which increases reliance on past crashes as a predictor of future crashes while apparently de-emphasizing known driver, vehicle, or safety management compliance issues. FMCSA told us that it incorporates a broad range of information including driver behavior, vehicle condition, and safety management in an attempt to capture and enable the agency to act on accident precursors in order to reduce crashes.

FMCSA is correct in concluding that the use of the negative binomial regression approach could tilt enforcement heavily toward carriers that have experienced crashes and away from other aspects of its problem areas, such as violation of vehicle safety standards, that are intended to prevent crashes. That is because the present SafeStat model does not statistically assign weights to the accident, driver, vehicle, and safety management areas. In addition, the negative binomial regression approach fully considers information on the results of driver and vehicle inspection data and safety management data. We used the same data that FMCSA used, with some adjustments as new information became available. While we found that the driver, vehicle, and safety management evaluation area scores are correlated with the future crash risk of a carrier, the accident evaluation area correlates the most with future crash risk. We recognize that FMCSA selects carriers for compliance reviews for multiple reasons, such as to respond to complaints, and we would expect that it would retain this flexibility if it adopted the negative binomial regression approach.

FMCSA also indicated that greater reliance on crash data increases emphasis on the least reliable available data set, and one that is out of the organization’s direct control—crash reporting. While our draft report found that crash reporting has improved, and that late reporting has not significantly impaired FMCSA’s use of the SafeStat model, FMCSA noted that the reliance on previous crashes in the negative binomial regression analysis described in our draft report could result in greater sensitivity to the crash data quality issues.

As FMCSA noted in its comments, our results showed that the effect of late-reported data was minimal. Also, as mentioned in our draft report and in this final report, it was not practical to determine the effect, if any, on SafeStat rankings of correcting inaccurate data or adding incomplete data. Since June 2004, FMCSA has devoted considerable efforts to improving the quality of the crash data it receives from the states. States are now tracked quarterly for the completeness, timeliness, and accuracy of their crash reporting. As FMCSA continues its efforts to have states improve these data, any sensitivity of results to crash data quality issues for the negative binomial regression approach should diminish.

We are sending copies of this report to congressional committees and subcommittees with responsibility for surface transportation safety issues; the Secretary of Transportation; the Administrator, FMCSA; and the Director, Office of Management and Budget. We also will make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions about this report, please either contact Sidney H. Schwartz at (202) 512-7387 or Susan A. Fleming at (202) 512-2834. Alternatively, they may be reached at schwartzsh@gao.gov or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff who made key contributions to this report are Carl Barden, Elizabeth Eisenstadt, Laurie Hamilton, Lisa Mirel, Stephanie Purcell, and James Ratzenberger.

Appendix I: Results of Other Assessments of the SafeStat Model’s Ability to Identify Motor Carriers That Pose High Crash Risks

Several studies by the Volpe National Transportation Systems Center (Volpe), the Department of Transportation’s Office of Inspector General, the Oak Ridge National Laboratory (Oak Ridge), and others have assessed the predictive capability of the Motor Carrier Safety Status Measurement System (SafeStat) model and the data used by that model. In general, those studies that assessed the predictive power of SafeStat offered suggestions to increase that power, and those studies that assessed data quality found weaknesses in the data that the Federal Motor Carrier Safety Administration (FMCSA) relies upon.

Assessments of SafeStat’s Predictive Capability

The studies we reviewed covered topics such as comparing SafeStat with random selection to determine which does a better job of selecting carriers that pose high crash risks, assessing whether statistical approaches could improve that selection, and analyzing whether carrier financial positions or driver convictions are associated with crash risk.

Predictive Capability of SafeStat Compared with Random Selection

In studies of the SafeStat model published in 2004 and 1998, Volpe analyzed retrospective data to determine how many crashes the carriers in SafeStat categories A and B experienced over the following 18 months. The 2004 study used the carrier rankings generated by the SafeStat model on March 24, 2001. Volpe then compared the SafeStat carrier safety ratings with state-reported data on crashes that occurred between March 25, 2001, and September 24, 2002, to assess the model’s performance. For each carrier, Volpe calculated a total number of crashes, weighted for time and severity, and then estimated a crash rate per 1,000 vehicles for comparing carriers in SafeStat categories A and B with the carriers in other SafeStat categories. The 1998 Volpe study used a similar methodology. Each study used a constrained subset of carriers rather than the full list contained in the Motor Carrier Management Information System (MCMIS). Both studies found that the crash rate for the carriers in SafeStat categories A and B was substantially higher than the other carriers during the 18 months after the respective SafeStat run. On the basis of this finding, Volpe concluded that the SafeStat model worked.

In response to a recommendation by the Department of Transportation’s Office of Inspector General, FMCSA contracted with Oak Ridge to independently review the SafeStat model. Oak Ridge assessed the SafeStat model’s performance and used the same data set (for March 24, 2001), provided by Volpe, that Volpe had used in its 2004 evaluation. Perhaps not surprisingly, Oak Ridge obtained a similar result for the weighted crash rate of carriers in SafeStat categories A and B over the 18-month follow-up period. As with the Volpe study, the Oak Ridge study was constrained because it was based on a limited data set rather than the entire MCMIS data set.

Application of Regression Models to Safety Data

While SafeStat does better than simple random selection in identifying carriers that pose high crash risks, other methods can also be used to achieve this outcome. Oak Ridge extended Volpe’s analysis by applying regression models to identify carriers that pose high crash risks. Specifically, Oak Ridge applied a Poisson regression model and a negative binomial model using the safety evaluation area values as independent variables to a weighted count of crashes that occurred in the 30 months before March 24, 2001. (For more information on statistical analyses, see app. III.)

In addition, Oak Ridge applied the empirical Bayes method to the negative binomial regression model and assessed the variability of carrier crash counts by estimating confidence intervals. Oak Ridge found that the negative binomial model worked well in identifying carriers that pose high crash risks. However, the data set Oak Ridge used did not include any carriers with one reported crash in the 30 months before March 24, 2001. Because data included only carriers with zero or two or more reported crashes, the distribution of crashes was truncated.

Since the Oak Ridge regression model analysis did not cover carriers with safety evaluation area data and one reported crash, the findings from the study are limited in their generalizability. However, other analyses of crashes at intersections and on road segments have also found that the negative binomial regression model works well. In addition, our analysis using a more recent and more comprehensive data set supports the finding that the negative binomial regression model performs better than the SafeStat model.

The studies carried out by other authors advocate the use of the empirical Bayes method in conjunction with a negative binomial regression model to estimate crash risk. Oak Ridge also applied this model to identify motor carriers that pose high crash risks. We applied this method to the 2004 SafeStat data and found that the empirical Bayes method best identified the carriers with the largest number of crashes in the 18 months after June 25, 2004. However, the crash rate per 1,000 vehicles was much lower than that for carriers in SafeStat categories A and B. We analyzed this result further and found that although the empirical Bayes method best identifies future crashes, it is not as effective as the SafeStat model or the negative binomial regression model in identifying carriers with the highest future crash rates. The carriers identified with the empirical Bayes method were invariably the largest carriers. This result is not especially useful from a regulatory perspective. Companies operating a large number of vehicles often have more crashes over a period of time than smaller companies. However, this does not mean that the larger company is necessarily violating more safety regulations or is less safe than the smaller company. For this reason, we do not advocate the use of the empirical Bayes method in conjunction with the negative binomial regression model as long as the method used to calculate the safety evaluation area values remains unchanged. If changes are made in how carriers are rated for safety, this method may in the future offer more promise than the negative binomial regression model alone.

Relationship of Carrier Financial Data and Safety Risk

Conducted on behalf of FMCSA, a study by Corsi, Barnard, and Gibney in 2002 examined how a carrier’s financial performance data correlate with the carrier’s score on a compliance review. The authors selected those motor carriers from MCMIS in December 2000 that had complete data for the accident, driver, vehicle, and safety management safety evaluation areas. Using these data, the authors then matched a total of 700 carriers to company financial statements in the annual report database of the American Trucking Associations. The authors created a binary response variable for whether the carrier received a satisfactory or an unsatisfactory outcome on the compliance review. The authors then assessed how this result correlated with financial measures derived from the company financial statements. In general, the study found that indicators of poor financial condition correlated with an increased safety risk.

Two practical considerations limit the applicability of the findings from this study to SafeStat. First, the 700 carriers in the study sample are not necessarily representative of the motor carriers that FMCSA oversees. Only about 2 percent of the carriers evaluated by the SafeStat model in June 2004 had a value for the safety management safety evaluation area. Of these carriers, not all had complete data for the other three safety evaluation areas. Second, FMCSA does not receive annual financial statements from all motor carriers. For these reasons, we did not consider using carrier financial data in our analysis of the SafeStat data.

Relationship of Commercial Driver License Convictions and Crash Risk

A series of studies by Lantz and others examined the effect of incorporating conviction data from the state-run commercial driver license data system into the calculation of a driver conviction measure. The studies found that the driver conviction measure is weakly correlated with the crash per vehicle rate. However, the studies did not incorporate the proposed driver conviction measure into one of the existing safety evaluation areas and use the updated measure to estimate new SafeStat scores for carriers. While the use of commercial driver license conviction data may have potential for future incorporation into a model for identifying carriers that pose high crash risks, there is no assessment of its impact at this time.

Impact of Data Quality on SafeStat’s Predictive Capability

The 2004 Office of Inspector General report, the 2004 Oak Ridge study, and reports by the University of Michigan Transportation Research Institute on state crash reporting all examined the impact of data quality on SafeStat’s ability to identify carriers that pose high crash risks. These studies looked at issues such as late reporting and incomplete or inaccurate reporting of crash data and found weaknesses.

Late Reporting of Crash Data

To determine whether states promptly report SafeStat data, the Office of Inspector General conducted a two-stage statistical sample in which it selected 10 states for review and then selected crash and inspection reports from those states for examination. It sampled 392 crash records and 400 inspection records from July through December 2002. In 2 of the 10 states selected, Pennsylvania and New Mexico, no crash records were available for the sample period, so it selected samples from earlier periods. The Office of Inspector General also discussed reporting issues with state and FMCSA officials and obtained crash records from selected motor carriers. In addition, the Office of Inspector General used the coefficient of variation to analyze data consistency and trends in reporting timeliness across geographic regions. Our review of the study indicates that it was based on sound audit methodology.

The study found that, as of November 2002, states submitted crash reports in fiscal year 2002 an average of 103 days after the crash occurred and that states varied widely in the timeliness of their crash data reporting. (FMCSA requires that states report crashes no more than 90 days after they occur.) In addition, the study found that 20 percent of the crashes that occurred in fiscal year 2002 were entered into MCMIS 6 months or more after the crash occurred. On the basis of this information, the Office of Inspector General concluded that the calculation of the accident safety evaluation area value was affected by the location of the carrier’s operations but did not estimate the degree of this effect.

We also assessed the extent of late reporting. We measured how many days, on average, it took each state to report crashes to MCMIS in each calendar year and found that the amount of time taken to report crashes declined from 2000 to 2005. Our findings were similar in nature to the Office of Inspector General’s findings. However, our results are broader because they are based on all crash data rather than a sample. In addition, since our work is more recent, it reflects more current conditions. We both came to the conclusion, although to varying degrees, that late reporting of crash data by states negatively affects SafeStat’s identification of carriers that pose high crash risks.

Oak Ridge also examined the impact of late reporting. Using data provided by Volpe, Oak Ridge looked at the difference between the date a crash occurred and the date it was entered into MCMIS. The researchers found that after 497 days, 90 percent of the reported crashes were entered into MCMIS.

The Oak Ridge study also reran the SafeStat model for March 2001 with the addition of crash data from March 2003 to see how more complete data changed SafeStat scores. The study found that the addition of late- reported data increased the number of carriers in the high-risk group by 18 percent. This late reporting affected the rankings of 8 percent of all the carriers ranked by SafeStat in March 2001. Of these affected carriers, 3 percent moved to a lower SafeStat category, and 5 percent moved to a higher category. Including the late-reported crash data available in March 2003 for the period from September 1998 through March 2001 resulted in a 35 percent increase in the available crash data.

We performed the same analysis as the Oak Ridge study and obtained similar results. We used SafeStat data from June 2004, which include carrier safety data from December 2001 through June 2004. Using FMCSA’s master crash file from June 2006, we found that, with the addition of late-reported crashes, 481 carriers moved into the highest risk category, and 182 carriers dropped out of the highest risk category resulting in a net increase of 299 carriers (6 percent) being added to the highest risk category.

The University of Michigan Transportation Research Institute issued a series of reports examining crash reporting rates in 14 states. These reports looked at late reporting as a potential source of low crash reporting rates but did not specifically examine the extent of late reporting or the impact of late reporting on SafeStat scores. The institute looked at reporting rates in each of the states by month to determine if reporting rates were lower in the latter part of the year because of late reporting. It found that reporting rates were lower in the latter part of the year in 6 of the 14 states studied. This issue was not a focus of our efforts, so we did not conduct a similar analysis.

Incomplete and Inaccurate Reporting of Crash Data

The Office of Inspector General’s study found several instances of incomplete or inaccurate data on crashes and carriers. The study reviewed MCMIS reporting for all states and found that 6 of them did not report any crashes to FMCSA in the 6-month period from July through December 2002. In addition, the study found that MCMIS listed about 11 percent of carriers as having no vehicles, and 15 percent as having no drivers. Finally, from a sample of crash records, the study estimated that 13 percent of the crash reports and 7 percent of the inspection reports in MCMIS contained errors that would affect SafeStat results. In particular, the study concluded that the database identified the wrong motor carrier as having been involved in a crash or as having received a violation in 11 percent of the erroneous records.

The University of Michigan Transportation Research Institute also examined the accuracy of states’ crash data reporting. To determine if crashes were reported accurately, the institute compared information contained in the individual states’ police accident reporting files with crash data reported to MCMIS. Some states, such as Ohio, had enough information captured in the police accident file to determine if individual crashes were eligible for reporting, and, therefore, the institute was able to use these data in its analyses. In other states, not enough information was available to make a determination, and the institute had to project results on the basis of other states’ experience. The institute also carried out a number of analyses, such as comparing reporting rates for different reporting jurisdictions, in an attempt to identify reporting trends in the individual states.

The institute identified several problems with the accuracy of states’ crash reporting. All 14 states that it studied reported ineligible crashes to MCMIS. These crashes were ineligible because they either involved vehicles not eligible for reporting or they did not meet the crash severity threshold. In total, the 14 states reported nearly 5,800 ineligible crashes to MCMIS out of almost 68,000 crashes reported (9 percent). The states also failed to report a number of eligible crashes: the 14 states studied reported from 9 percent to 87 percent of eligible crashes.

Our review of the institute’s methodology indicates that its findings are based on sound methodology and that its analyses were very thorough. However, its studies are limited to the 14 states studied and to the particular year studied. (Not all studies covered the same year.) These states’ experience may or may not be representative of the experiences of the entire country, and there is no way to determine if the reporting for this year is representative of the state’s reporting activities over a number of years or if the results were unique to that particular year. The exceptions to this are the studies for Missouri, which covered calendar years 2001 and 2005, and Ohio, which covered calendar years 2000 and 2005.

We did not attempt to assess the extent of inaccurate reporting in individual states, but we did find examples of inaccurate data reporting. To analyze the completeness of reporting, we attempted to match all crash records in the MCMIS master crash file for crashes occurring between December 26, 2001, and June 25, 2004, to the list of motor carriers in the MCMIS census file. We found that Department of Transportation numbers were missing for 30 percent of the crashes that were reported, and the number did not match a Department of Transportation number listed in MCMIS for 8 percent of reported crashes. We also compared the number of crashes in MCMIS with the number in the General Estimates System produced by the National Highway Traffic Safety Administration and found evidence of underreporting of crashes to MCMIS.

Appendix II: Scope and Methodology

To determine whether statistical approaches could be used to improve FMCSA’s ability to identify carriers that pose high crash risks, we tested a variety of regression models and compared their results with results from the existing SafeStat model. The models we tested, using MCMIS data used by SafeStat in June 2004 to identify carriers that pose high crash risks, include the Poisson, negative binomial, zero-inflated negative binomial, zero-inflated Poisson, and empirical Bayes. We chose these regression models because crash totals for a company represent count outcomes, and these statistical models are appropriate for use with count data. In addition, we explored logistic regression to assess the odds of having a crash. Based on the results of the statistical models, we ranked the predicted means (or predicted probabilities in the logistic regression) to see which carriers would be at risk during the 18-month period after June 2004. We selected June 2004 because this date enabled us to examine MCMIS data on actual crashes that occurred in the 18-month period from July 2004 through December 2005. We used these data to determine the degree to which SafeStat identified carriers that proved to pose high crash risks. We then compared the predictive performance of the regression models with the performance of SafeStat to determine which method best identified carriers that pose high crash risks. Using a series of simple random samples, we also calculated the crash rates of all carriers listed in the main SafeStat summary results table in MCMIS for comparison with the crash rates of carriers identified by SafeStat as high risk. We did this analysis to determine whether the SafeStat model did a better job than random selection of identifying motor carriers that pose high crash risks.

In addition, we tested changes to selected portions of the SafeStat model to see whether improvements could be made in the identification of high- risk motor carriers. In one analysis, we modified the calculation of the safety evaluation area values and compared the number of high-risk motor carriers identified with the number identified by the unmodified safety evaluation areas. For example, we included carriers with only one crash in the calculation of the accident safety evaluation area whereas the unmodified SafeStat model includes only carriers with two or more crashes. We also investigated the effect of removing the time and severity weights from the indexes used to construct the accident, driver, and vehicle safety evaluation areas. We then compared the result of using the modified and unmodified safety evaluation area values to determine if this modification improved the model’s ability to identify future crash risks.

To assess the extent to which the timeliness, completeness, and accuracy of MCMIS and state-reported crash data affect SafeStat’s performance, we carried out a series of analyses with the MCMIS crash master file and MCMIS census file, as well as surveying the literature to assess findings on MCMIS data quality from other studies. To assess the effect of timeliness, we first measured how many days on average it was taking each state to report crashes to FMCSA by year for calendar years 2000 through 2005. We also recalculated SafeStat scores from the model’s June 25, 2004, run to include crashes that had occurred more than 90 days before that date but had not been reported to FMCSA by that date. We compared the number and rankings of carriers from the original SafeStat results with those obtained by adding in data for the late-reported crashes. In addition, we reviewed the University of Michigan Transportation Research Institute’s studies of state crash reporting to MCMIS to identify the impact of late reporting in individual states on MCMIS data quality.

To assess the effect of completeness, we attempted to match all crash records in the MCMIS crash file for crashes occurring from December 2001 through June 2004 to the list of motor carriers in the MCMIS census file. In addition, we reviewed the University of Michigan Transportation Research Institute’s studies of state crash reporting to MCMIS to identify the impact of incomplete crash reporting in individual states on MCMIS data quality.

To assess the effect of accuracy, we reviewed a report by the Office of Inspector General that tested the accuracy of electronic data by comparing records selected in the sample with source paper documents. In addition, we reviewed the University of Michigan Transportation Research Institute’s studies of state crash reporting to MCMIS to identify the impact of incorrectly reported crashes in individual states on MCMIS data quality.

While the limitations in the data adversely affect the ability of any method to identify carriers that pose high crash risks, we determined that the data were of sufficient quality for our use, which was to assess how the application of regression models might improve the ability to identify high- risk carriers over the current approach—not to determine absolute measures of crash risk. Our reasoning is based on the fact that we used the same data set to compare the results of the SafeStat model and the regression models. Limitations in the data would apply equally to both results. Methods to identify carriers that pose high crash risk will perform more efficiently once the known problems with the quality of state- reported crash data are addressed.

To understand what other researchers have found about how well SafeStat identifies motor carriers that pose high crash risks, we identified studies through a general literature review and by asking stakeholders and study authors to identify high-quality studies. Studies included in our review were (1) the 2004 study of SafeStat done by Oak Ridge National Laboratory, (2) the SafeStat effectiveness studies done by the Department of Transportation Office of Inspector General and Volpe Institute, (3) the University of Michigan Transportation Research Institute’s studies of state crash reporting to FMCSA, and (4) the 2006 Department of Transportation Office of Inspector General’s audit of data for new entrant carriers. We assessed the methodology used in each study and identified which findings are supported by rigorous analysis. We accomplished this by relying on information presented in the studies and, where possible, by discussing the studies with the authors. When the studies’ methodologies and analyses appeared reasonable, we used those findings in our analysis of SafeStat. We discussed with FMCSA and industry and safety stakeholders the SafeStat methodology issues and data quality issues raised by these studies. We also discussed the aptness of the respective methodological approaches with FMCSA. Finally, we reviewed FMCSA documentation on how SafeStat is constructed and assessments of SafeStat conducted by FMCSA.

Appendix III: Additional Results from Our Statistical Analyses of the SafeStat Model

This appendix contains technical descriptions and other information related to our statistical analyses.

Overview of Regression Analyses

To study how well statistical methods identify carriers that pose high crash risks, we carried out a series of regression analyses. The safety evaluation area values for the accident, driver, vehicle, and safety management areas served as the independent variables to predict crash risks. We used the state-reported crash data in MCMIS for crashes that occurred during the 30 months preceding June 25, 2004, as the dependent variable in each model. We used the results of the SafeStat model run from June 25, 2004, to benchmark the performance of the regression models with the crash records for the identified high-risk carriers over the succeeding 18 months.

We matched the state-reported crashes that occurred from December 26, 2001, through June 25, 2004, to the carriers listed in SafeStat. We checked our match of crashes for carriers with those carriers used by FMCSA in June 2004 and found that the reported numbers had changed for about 10,700 carriers in the intervening 2 years. We found this difference even though we used only crashes that occurred from December 26, 2001, through June 25, 2004, and were reported to FMCSA before June 25, 2004. Because of this difference in matched crashes, we recalculated the accident safety evaluation area using our match of the crashes. This is discussed later in more detail.

Using our recalculation of the accident safety evaluation area values and the original driver, vehicle, and safety management safety evaluation area values for the carriers, we fit a Poisson regression model and a negative binomial regression model to the crash counts. Both of these models are statistically appropriate for use when modeling counts that are positive and integer valued. The two models differ in their assumptions about the mean and variance. Whereas the Poisson model assumes that the mean and the variance are equal, the negative binomial model assumes the mean is not equal to the variance. The crash data in MCMIS fit the assumptions of the negative binomial distribution better than those of the Poisson.

We also tried to estimate zero-inflated Poisson and zero-inflated negative binomial models with the SafeStat data. These models are appropriate when the count values include many zeros, as is the case with the values in this data set (because many carriers do not have crash records). However, we could not estimate the parameters for these models with the MCMIS data. We also considered using logistic regression to model the carrier’s odds of experiencing a crash. However, the parameter estimates of the four safety evaluation area values could not be estimated, so we did not use the results of this model.

Finally, we used the results from the negative binomial model to assess the expected carrier crash counts using the empirical Bayes estimate. In safety applications, the empirical Bayes method is used to increase the precision of estimates and correct for the regression-to-mean bias. In this application, the empirical Bayes method calculates a weighted average of the rate of crashes for a carrier from the prior 30 months with the predicted mean number of crashes from the negative binomial regression. This method optimizes the identification of carriers with the highest number of future crashes. This optimization of total crashes, however, resulted in the identification of primarily the largest companies. The crash rate (crashes per 1,000 vehicles per 18 months) was not as high for this group as for the carriers placed by the SafeStat model in its A and B categories.

Technical Explanation of the Negative Binomial Regression Model

This section provides the technical details for the negative binomial regression model fit to the SafeStat data. This section also explains how we handled incomplete safety evaluation area data for carriers in the regression model analyses. ) ) ) exp( β ) )

This equation models the log of the expected mean number of crashes for each motor carrier using the four safety evaluation area values, but most commercial motor companies listed in MCMIS do not have values for all four safety evaluation areas. To account for this, it is necessary to define four indicator variables. Let . β ) ) β ) )

A carrier has to have two or more reported crashes in the past 30 months to receive an accident safety evaluation area value. A carrier has to have three or more roadside inspections to receive a driver or vehicle safety evaluation area value. A driver has to have had a compliance review in the past 18 months to receive a safety management safety evaluation area value. There are other ways a carrier can receive a value for one of these four safety evaluation areas, refer to the description of each one provided in the Background. intercept for the indicator term (the coefficient for the indicator function).

We used a similar parameterization to formulate the Poisson regression model.

Evaluation of Regression Models’ Performance

We estimated regression models using the same data FMCSA used in its application of the SafeStat model on June 25, 2004, with one exception for the accident safety evaluation area. For that area, we used our own match of crashes to carriers for December 26, 2001, through June 25, 2004. The MCMIS data we received in June 2006 produced different totals in the match of crashes to carriers for about 10,700 carriers. MCMIS data change over time because crash data are added, deleted, or changed as more information about these crashes is obtained. The discrepancies in matching arose even though we used the identical time interval and counted crashes only when the record indicated they had been reported to FMCSA before June 25, 2004. Because of these discrepancies, it was necessary to calculate the accident safety evaluation area values using our match of crashes and then recalculate the SafeStat carrier scores for June 25, 2004, using our accident safety evaluation area values and the original driver, vehicle, and safety management safety evaluation area values. We used our accident safety evaluation area values and the original driver, vehicle, and safety management safety evaluation area values in the regression model analysis.

Using the revised accident safety evaluation area values and FMCSA’s original driver, vehicle, and safety management safety evaluation area values, the SafeStat model identified 4,989 carriers that pose high crash risks. For each regression model, we input the safety evaluation area data for the carriers in our analysis data set and used the regression model to calculate the predicted mean number of crashes. We then sorted the predicted scores and selected the 4,989 carriers with the worst predicted values as the set of high-risk carriers identified by the regression model. Next, we used MCMIS to determine the crash history of these 4,989 carriers between June 26, 2004, and December 25, 2005, and compared the aggregate crash history with the aggregate crash history of the carriers identified by the SafeStat model during the same period of time.

The regression models do not categorize carriers by letter; the regression models produce a predicted crash risk for each carrier. The regression models make use of the safety evaluation area values, but they differ from the SafeStat model in this respect.

The results show that a negative binomial regression model estimated with the safety evaluation area values outperforms the current SafeStat model in terms of predicting future crashes and the future crash rate among identified carriers that pose high crash risks. (See table 3.) That is, our negative binomial and Poisson models show 111 and 109 crashes per 1,000 vehicles per 18 months, respectively, compared with the 102 crashes per 1,000 vehicles per 18 months estimated by the current SafeStat model. The Poisson model is not as appropriate since the crash counts for carriers have variability that is significantly different from the mean number of crashes. The empirical Bayes method optimizes the selection of future crashes; however, it does so by selecting the largest carriers. The largest carriers have a lower crash rate per 1,000 vehicles per 18 months than the carriers that pose high crash risks identified by the SafeStat model or by the negative binomial regression model. Since the primary use of SafeStat is to identify and prioritize carriers for FMCSA and state compliance reviews, the empirical Bayes method did not identify carriers with the highest safety risk.