Motor Carrier Safety: Improvements to Data-Driven Oversight Could Better Target High Risk Carriers
Released: 2015-03-04T00:00:00

Letter

FMCSA’s Method Does Not Effectively Identify High Risk Carriers

As we reported in our February 2014 report, since CSA was implemented nationwide in 2010, it has been successful in raising the profile of safety in the motor carrier industry and providing FMCSA with more tools to increase interventions with carriers. We found that following the implementation of CSA, FMCSA was potentially able to reach a larger number of carriers, primarily by sending them warning letters. Law enforcement officials and industry stakeholders we interviewed generally supported the structure of the CSA program, in part because CSA provides data about the safety record of individual carriers, such as data on inspections, violations, crashes, and investigations, that help guide the work of state inspectors during inspections. However, despite these advantages, our report also uncovered major challenges in reliably assessing safety risk and targeting the riskiest carriers.

First, according to FMCSA, SMS was designed to use all safety-related violations of FMCSA regulations recorded during roadside inspections. For SMS to be effective in identifying carriers at risk of crashing, the violation information that is used to calculate SMS scores should have a relationship with crash risk. However, we found that the relationship between the violation of most of these regulations and crash risk is unclear, potentially limiting the effectiveness of SMS in identifying carriers that are likely to crash. Our analysis found that most of the safety regulations used in SMS were violated too infrequently over a 2-year period to reliably assess whether they were accurate predictors of an individual carrier’s likelihood to crash. Specifically, we found that 593 of the 754 regulations we examined were violated by less than one percent of carriers. Of the remaining regulations with sufficient violation data, we found 13 regulations for which violations consistently had some association with crash risk in at least half the tests we performed, and only two regulations had sufficient data to consistently establish a substantial and statistically reliable relationship with crash risk across all of our tests.

Second, most carriers lack sufficient safety performance data, such as information from inspections, to ensure that FMCSA can reliably compare them with other carriers. SMS scores are based on violation rates that are calculated by dividing a carrier’s violations by either the number of inspections or vehicles associated with a carrier. The precision and reliability of these rates varies greatly depending on the number of inspections or vehicles a carrier has. Violation rates calculated for carriers with more inspections or vehicles will have more precision and confidence than those with only a few inspections or vehicles. This statistical reality is critical to SMS, because for the majority of the industry, the number of inspections or vehicles for an individual carrier is very low. About two- thirds of carriers we evaluated operated fewer than four vehicles and more than 93 percent operated fewer than 20 vehicles. Moreover, many of these carriers’ vehicles were inspected infrequently. Carriers with few inspections or vehicles will potentially have estimated violation rates that are artificially high or low and thus not sufficiently precise for comparison across carriers. This creates the likelihood that many SMS scores do not accurately or precisely assess safety for a specific carrier. FMCSA acknowledged that violation rates for carriers with few inspections or vehicles can be less precise, but the methods FMCSA uses to address this limitation are not effective. For example, FMCSA requires a minimum level of data (i.e., inspections or violations) for a carrier to receive an SMS score. However, we found that level of data is not sufficient to ensure reliable results.

Our analysis of the effectiveness of FMCSA’s existing CSA methodology found that the majority of the carriers that SMS identified as having the highest risk for crashing in the future did not actually crash. Moreover, smaller carriers and carriers with few inspections or vehicles tended to be disproportionately targeted for intervention. As a result, FMCSA may devote intervention resources to carriers that do not necessarily pose as great a safety risk as other carriers. In our 2014 report, we illustrated that when SMS only considered carriers with more safety information, such as inspections, it was better able to identify carriers that later crashed and allowed for better targeting of resources. An approach like this would involve trade-offs; fewer carriers would receive SMS scores, but these scores would generally be more reliable for targeting FMCSA’s intervention resources. FMCSA could still use the safety information available to oversee the remaining carriers the same way it currently oversees the approximately 72 percent of carriers that do not receive SMS scores using its existing approach.

Given the limitations of safety performance information, we concluded that it is important that FMCSA consider how reliable and precise SMS scores need to be for the purposes for which they are used. FMCSA reports these scores publicly and is considering using a carrier’s performance information to determine its fitness to operate. FMCSA includes a disclaimer with the publicly released SMS scores, which states that the data are intended for agency and law enforcement purposes, and that readers should draw conclusions about a carrier’s safety condition based on the carrier’s official safety rating rather than its SMS score. At the same time, FMCSA has also stated that SMS provides stakeholders with valuable safety information, which can “empower motor carriers and other stakeholders…to make safety-based business decisions.” As a result, some stakeholders we spoke to, such as industry and law enforcement groups, have said that there is a lot of confusion in the industry about what the SMS scores mean and that the public, unlike law enforcement, may not understand the limitations of the system.

Based on the concerns listed above, in our 2014 report we recommended that FMCSA revise the SMS methodology to better account for limitations in available information when drawing comparisons of safety performance across carriers. We further recommended that FMCSA’s determination of a carrier’s fitness to operate should account for limitations we identified regarding safety performance information. FMCSA did not concur with our recommendation to revise the SMS methodology because, according to FMCSA officials, SMS in its current state sufficiently prioritizes carriers for intervention purposes. However, FMCSA agreed with our recommendation on the determination of a carrier’s fitness to operate, but has not yet taken any actions. As I will discuss later in my statement, we continue to believe that FMCSA should improve its SMS methodology.

FMCSA Cannot Readily Determine the Number of Chameleon Carriers

As we reported in our March 2012 report, FMCSA also faces significant challenges in determining the prevalence of chameleon carriers, in part, because there are approximately 75,000 new applicants each year. As mentioned earlier, chameleon carriers are motor carriers disguising their former identity to evade enforcement actions. FMCSA has established a vetting program to review each new application for operating authority submitted by passenger carriers (intercity and charter or tour bus operators) and household goods carriers (hired by consumers to move personal property). According to FMCSA officials, FMCSA vetted all applicants in these groups for two reasons: (1) these two groups pose higher safety and consumer protection concerns than other carrier groups and (2) it does not have the resources to vet all new carriers. While FMCSA’s exclusive focus on passenger and household goods carriers limits the vetting program to a manageable number, it does not account for the risk presented by chameleon carriers in the other groups, such as for-hire freight carriers, that made up 98 percent of new applicants in 2010.

We found that using data analysis to target new applicants would allow FMCSA to expand its examinations of newly registered carriers to include new applicants of all types using few or no additional staff resources. Our analysis of FMCSA data found that 1,136 new motor carrier applicants in 2010 had chameleon attributes, of which 1,082 were freight carriers.Even with the large number of new applicant carriers and constraints on its resources, we concluded in 2012 that FMCSA could target the carriers that present the highest risk of becoming chameleon carriers by using a data-driven, risk-based approach.

As a result of these findings, we recommended that FMCSA use a data- driven, risk-based approach to target carriers at high risk for becoming chameleon carriers. This would allow expansion of the vetting program to all carriers with chameleon attributes, including freight carriers. FMCSA agreed with our recommendations. In June 2013, to help better identify chameleon carriers, FMCSA developed and began testing a risk-based methodology that implemented a framework that closely follows the methodology we discussed in our report. FMCSA’s preliminary analysis of this methodology indicates that it is generally successful in providing a risk-based screening of new applicants, which it plans to use as a front- end screening methodology for all carrier types seeking operating authority. By developing this risk-based methodology and analyzing the initial results, FMCSA has developed an approach that may help keep unsafe carriers off the road.

To further help Congress with its oversight of FMCSA and motor carrier safety, we also have on-going work on FMCSA’s hours-of-service regulations, DOD’s Transportation Protective Services program, and commercial driver’s licenses. This work is in various stages, and we expect to issue the final reports later this year.

In conclusion, the commercial motor carrier industry is large and dynamic, and FMCSA plays an important role in identifying and removing unsafe carriers from the roadways. With over 500,000 active motor carriers, it is essential to examine ways to better target FMCSA’s resources to motor carriers presenting the greatest risk. To effectively do this, FMCSA must use a number of strategies to identify and intervene with high risk carriers. We continue to believe that a data-driven, risk-based approach for identifying high risk carriers holds promise. FMCSA’s preliminary steps to implement a risk-based screening methodology have the potential to identify more high risk chameleon carriers. However, without efforts to revise its SMS methodology, FMCSA will not be able to effectively target its intervention resources toward the highest risk carriers and will be challenged to meet its mission of reducing the overall crashes, injuries, and fatalities involving large trucks and buses.

Chairwoman Fischer, Ranking Member Booker, and Members of the Subcommittee, this concludes my prepared remarks. I would be pleased to answer any questions you or other Members may have at this time.

GAO Contact and Staff Acknowledgements

For further information regarding this statement, please contact Susan Fleming at (202) 512-2834 or Flemings@gao.gov about this statement. Contact points for our Offices of Congressional Relations and Public Relations can be found on the last page of this statement. Matt Cook, Jen DuBord, Sarah Farkas, Brandon Haller, Matt LaTour, and Amy Rosewarne made key contributions to this statement.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.