Motor Carrier Safety: Improvements to Data-Driven Oversight Could Better Target High Risk Carriers
Published: 2015-03-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: FMCSA's primary mission of reducing crashes, injuries, and fatalities involving large trucks and buses is critical to the safety of our nation's highways. However, with more than 500,000 active motor carriers operating on U.S. roadways, FMCSA must screen, identify, and target its resources toward those carriers presenting the greatest risk for crashing in the future.
FMCSA has recently taken some steps in this direction by, among other actions:
Establishing its oversight program—the CSA program—based on a data-driven approach for identifying motor carriers at risk of presenting a safety hazard or causing a crash, and
Establishing a vetting program designed to detect potential “chameleon” carriers—those carriers that have deliberately disguised their identity to evade enforcement actions issued against them.
This testimony provides information on both of these programs, based on two recent GAO reports on the oversight challenges FMCSA faces in identifying high risk motor carriers for intervention ( GAO-14-114 ), and chameleon carriers ( GAO-12-364 ), respectively.
section_title: What GAO Found
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA) has taken steps toward better oversight of motor carriers by establishing the Compliance, Safety, Accountability (CSA) and chameleon carrier vetting programs; however, FMCSA could improve its oversight to better target high risk carriers. The CSA program oversees carriers' safety performance through roadside inspections and crash investigations, and issues violations when instances of noncompliance with safety regulations are found. CSA provides FMCSA, state safety authorities, and the industry with valuable information regarding carriers' performance on the road.
A key component of CSA—the Safety Measurement System (SMS)—uses carrier performance data collected from inspections and investigations to calculate safety scores for carriers and identify those at high risk of causing a crash. The program then uses these scores to target high risk carriers for enforcement actions, such as warning letters, additional investigations, or fines. However, GAO's 2014 report identified two major challenges that limit the precision of the SMS scores and confidence that these scores are effectively comparing safety performance across carriers.
First, SMS uses violations of safety-related regulations to calculate a score, but GAO found that most of these regulations were violated too infrequently to determine whether they were accurate predictors of crash risk. Second, most carriers lacked sufficient data from inspections and violations to ensure that a carrier's SMS score could be reliably compared with scores for other carriers. GAO concluded that these challenges raise questions about whether FMCSA is able to identify and target the carriers at highest risk for crashing in the future. To address these challenges, GAO recommended, among other things, that FMCSA revise the SMS methodology to better account for limitations in available information when drawing comparisons of safety performance across carriers. FMCSA did not concur with GAO's recommendation to revise the SMS methodology because it believed that SMS sufficiently prioritized carriers for intervention. Therefore, FMCSA has not taken any actions.
GAO continues to believe that a data-driven, risk-based approach holds promise, and efforts to improve FMCSA's oversight could allow it to more effectively target its resources toward the highest risk carriers, and better meet its mission of reducing the overall crashes, injuries, and fatalities involving motor carriers.
GAO's 2012 report found that FMCSA examined only passenger and household goods carriers as part of its chameleon carrier vetting program for new applicants. GAO found that by modifying FMCSA's vetting program, FMCSA could expand its examinations of newly registered carriers to include all types of carriers, including freight carriers, using few additional staff resources. GAO recommended that FMCSA develop, implement, and evaluate the effectiveness of a data-driven, risk-based vetting methodology to target carriers with chameleon attributes. FMCSA concurred with GAO's recommendation and has taken actions to address these recommendations.

section_title: Letter
paragraphs: I am pleased to be here today to discuss oversight of the U.S.  Department of Transportation’s (USDOT) Federal Motor Carrier Safety  Administration (FMCSA). The commercial motor carrier industry is large  and diverse, with more than 500,000 active motor carriers operating on  U.S. roadways. FMCSA’s primary mission of reducing crashes, injuries,  and fatalities involving large trucks and buses is critical to the safety of  our nation’s highways. To accomplish this mission, FMCSA engages in a  range of activities designed to screen, identify, and target its resources  toward the motor carriers that demonstrate characteristics or behaviors  that increase the risk of crashing. Among these activities are new entrant  safety audits and identification, or vetting, of “chameleon” carriers—motor  carriers that have registered and been operating illegally in interstate  commerce by using a new identity in an effort to disguise their former  identity and evade enforcement actions issued against them by FMCSA.
FMCSA’s oversight program—the Compliance, Safety, Accountability  (CSA) program—is based on the Safety Measurement System (SMS), a  data-driven approach for identifying motor carriers at risk of presenting a  safety hazard or causing a crash. SMS uses information collected during  federal and state roadside inspections and from reported crashes to  calculate scores across seven categories that quantify a carrier’s safety  performance relative to other carriers. The precision and accuracy of  these scores is vital because FMCSA investigators and their state  partners use SMS results to focus their resources on higher risk carriers  and, through interventions, help reduce the number of motor carrier  crashes, injuries, and fatalities. FMCSA currently posts most of the scores  publicly on its website for use by industry stakeholders and the public.FMCSA has indicated that a future rulemaking will include some of the  information used to calculate SMS scores to help determine a carrier’s  overall fitness to operate motor vehicles.
My statement today presents highlights from our two recent reports on the  oversight challenges FMCSA faces in identifying high risk motor carriers.
Each of these reports contains detailed information on our objectives,  scope, and methodology for performing this work. The work on which this  statement is based was performed in accordance with generally accepted  government auditing standards. Those standards require that we plan  and perform the audit to obtain sufficient, appropriate evidence to provide  a reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
In summary, our recent work on FMCSA oversight found that the  establishment of the CSA program and chameleon carrier vetting  program are steps toward better oversight of motor carriers. Through the  CSA program, FMCSA can provide the agency, state safety authorities,  and the industry with valuable information regarding carriers’ performance  on the road and reach more carriers through interventions. However,  while we continue to believe that a data-driven, risk-based approach  holds promise, our work identified several major challenges that limit the  precision of SMS scores and confidence that these scores are effectively  comparing safety performance across carriers. These serious challenges  raise questions about whether CSA is able to identify and target the  carriers at highest risk for crashing in the future. In addition, our recent  work on FMCSA’s chameleon carrier vetting program found that using  data analysis for targeting new applicants would allow FMCSA to expand  its examinations of newly registered carriers to include new applicants of  all types rather than just passenger and household goods carriers, using  few or no additional staff resources. FMCSA has taken actions to address  our recommendations related to the vetting of chameleon carriers;  however, it has not taken action to address our recommendations to  better account for limitations in the CSA program.
subsections: section_title: FMCSA’s Method Does Not Effectively Identify High Risk Carriers
paragraphs: As we reported in our February 2014 report, since CSA was implemented  nationwide in 2010, it has been successful in raising the profile of safety  in the motor carrier industry and providing FMCSA with more tools to  increase interventions with carriers. We found that following the  implementation of CSA, FMCSA was potentially able to reach a larger  number of carriers, primarily by sending them warning letters. Law  enforcement officials and industry stakeholders we interviewed generally  supported the structure of the CSA program, in part because CSA  provides data about the safety record of individual carriers, such as data  on inspections, violations, crashes, and investigations, that help guide the  work of state inspectors during inspections. However, despite these  advantages, our report also uncovered major challenges in reliably  assessing safety risk and targeting the riskiest carriers.
First, according to FMCSA, SMS was designed to use all safety-related  violations of FMCSA regulations recorded during roadside inspections.  For SMS to be effective in identifying carriers at risk of crashing, the  violation information that is used to calculate SMS scores should have a  relationship with crash risk. However, we found that the relationship  between the violation of most of these regulations and crash risk is  unclear, potentially limiting the effectiveness of SMS in identifying carriers  that are likely to crash. Our analysis found that most of the safety  regulations used in SMS were violated too infrequently over a 2-year  period to reliably assess whether they were accurate predictors of an  individual carrier’s likelihood to crash. Specifically, we found that 593 of  the 754 regulations we examined were violated by less than one percent  of carriers. Of the remaining regulations with sufficient violation data, we  found 13 regulations for which violations consistently had some  association with crash risk in at least half the tests we performed, and  only two regulations had sufficient data to consistently establish a  substantial and statistically reliable relationship with crash risk across all  of our tests.
Second, most carriers lack sufficient safety performance data, such as  information from inspections, to ensure that FMCSA can reliably compare  them with other carriers. SMS scores are based on violation rates that are  calculated by dividing a carrier’s violations by either the number of  inspections or vehicles associated with a carrier. The precision and  reliability of these rates varies greatly depending on the number of  inspections or vehicles a carrier has. Violation rates calculated for carriers  with more inspections or vehicles will have more precision and confidence  than those with only a few inspections or vehicles. This statistical reality  is critical to SMS, because for the majority of the industry, the number of  inspections or vehicles for an individual carrier is very low. About two- thirds of carriers we evaluated operated fewer than four vehicles and  more than 93 percent operated fewer than 20 vehicles. Moreover, many  of these carriers’ vehicles were inspected infrequently. Carriers with few  inspections or vehicles will potentially have estimated violation rates that  are artificially high or low and thus not sufficiently precise for comparison  across carriers. This creates the likelihood that many SMS scores do not  accurately or precisely assess safety for a specific carrier. FMCSA  acknowledged that violation rates for carriers with few inspections or  vehicles can be less precise, but the methods FMCSA uses to address  this limitation are not effective. For example, FMCSA requires a minimum  level of data (i.e., inspections or violations) for a carrier to receive an SMS  score. However, we found that level of data is not sufficient to ensure  reliable results.
Our analysis of the effectiveness of FMCSA’s existing CSA methodology  found that the majority of the carriers that SMS identified as having the  highest risk for crashing in the future did not actually crash. Moreover,  smaller carriers and carriers with few inspections or vehicles tended to be  disproportionately targeted for intervention. As a result, FMCSA may  devote intervention resources to carriers that do not necessarily pose as  great a safety risk as other carriers. In our 2014 report, we illustrated that  when SMS only considered carriers with more safety information, such as  inspections, it was better able to identify carriers that later crashed and  allowed for better targeting of resources. An approach like this would  involve trade-offs; fewer carriers would receive SMS scores, but these  scores would generally be more reliable for targeting FMCSA’s  intervention resources. FMCSA could still use the safety information  available to oversee the remaining carriers the same way it currently  oversees the approximately 72 percent of carriers that do not receive  SMS scores using its existing approach.
Given the limitations of safety performance information, we concluded  that it is important that FMCSA consider how reliable and precise SMS  scores need to be for the purposes for which they are used. FMCSA  reports these scores publicly and is considering using a carrier’s  performance information to determine its fitness to operate. FMCSA  includes a disclaimer with the publicly released SMS scores, which states  that the data are intended for agency and law enforcement purposes, and  that readers should draw conclusions about a carrier’s safety condition  based on the carrier’s official safety rating rather than its SMS score. At  the same time, FMCSA has also stated that SMS provides stakeholders  with valuable safety information, which can “empower motor carriers and  other stakeholders…to make safety-based business decisions.” As a  result, some stakeholders we spoke to, such as industry and law  enforcement groups, have said that there is a lot of confusion in the  industry about what the SMS scores mean and that the public, unlike law  enforcement, may not understand the limitations of the system.
Based on the concerns listed above, in our 2014 report we recommended  that FMCSA revise the SMS methodology to better account for limitations  in available information when drawing comparisons of safety performance  across carriers. We further recommended that FMCSA’s determination of  a carrier’s fitness to operate should account for limitations we identified  regarding safety performance information. FMCSA did not concur with our  recommendation to revise the SMS methodology because, according to  FMCSA officials, SMS in its current state sufficiently prioritizes carriers for  intervention purposes. However, FMCSA agreed with our  recommendation on the determination of a carrier’s fitness to operate, but  has not yet taken any actions. As I will discuss later in my statement, we  continue to believe that FMCSA should improve its SMS methodology.
section_title: FMCSA Cannot Readily Determine the Number of Chameleon Carriers
paragraphs: As we reported in our March 2012 report, FMCSA also faces significant  challenges in determining the prevalence of chameleon carriers, in part,  because there are approximately 75,000 new applicants each year. As  mentioned earlier, chameleon carriers are motor carriers disguising their  former identity to evade enforcement actions. FMCSA has established a  vetting program to review each new application for operating authority  submitted by passenger carriers (intercity and charter or tour bus  operators) and household goods carriers (hired by consumers to move  personal property). According to FMCSA officials, FMCSA vetted all  applicants in these groups for two reasons: (1) these two groups pose  higher safety and consumer protection concerns than other carrier groups  and (2) it does not have the resources to vet all new carriers. While  FMCSA’s exclusive focus on passenger and household goods carriers  limits the vetting program to a manageable number, it does not account  for the risk presented by chameleon carriers in the other groups, such as  for-hire freight carriers, that made up 98 percent of new applicants in  2010.
We found that using data analysis to target new applicants would allow  FMCSA to expand its examinations of newly registered carriers to include  new applicants of all types using few or no additional staff resources. Our  analysis of FMCSA data found that 1,136 new motor carrier applicants in  2010 had chameleon attributes, of which 1,082 were freight carriers.Even with the large number of new applicant carriers and constraints on  its resources, we concluded in 2012 that FMCSA could target the carriers  that present the highest risk of becoming chameleon carriers by using a  data-driven, risk-based approach.
As a result of these findings, we recommended that FMCSA use a data- driven, risk-based approach to target carriers at high risk for becoming  chameleon carriers. This would allow expansion of the vetting program to  all carriers with chameleon attributes, including freight carriers. FMCSA  agreed with our recommendations. In June 2013, to help better identify  chameleon carriers, FMCSA developed and began testing a risk-based  methodology that implemented a framework that closely follows the  methodology we discussed in our report. FMCSA’s preliminary analysis of  this methodology indicates that it is generally successful in providing a  risk-based screening of new applicants, which it plans to use as a front- end screening methodology for all carrier types seeking operating  authority. By developing this risk-based methodology and analyzing the  initial results, FMCSA has developed an approach that may help keep  unsafe carriers off the road.
To further help Congress with its oversight of FMCSA and motor carrier  safety, we also have on-going work on FMCSA’s hours-of-service  regulations, DOD’s Transportation Protective Services program, and  commercial driver’s licenses. This work is in various stages, and we  expect to issue the final reports later this year.
In conclusion, the commercial motor carrier industry is large and dynamic,  and FMCSA plays an important role in identifying and removing unsafe  carriers from the roadways. With over 500,000 active motor carriers, it is  essential to examine ways to better target FMCSA’s resources to motor  carriers presenting the greatest risk. To effectively do this, FMCSA must  use a number of strategies to identify and intervene with high risk carriers.  We continue to believe that a data-driven, risk-based approach for  identifying high risk carriers holds promise. FMCSA’s preliminary steps to  implement a risk-based screening methodology have the potential to  identify more high risk chameleon carriers. However, without efforts to  revise its SMS methodology, FMCSA will not be able to effectively target  its intervention resources toward the highest risk carriers and will be  challenged to meet its mission of reducing the overall crashes, injuries,  and fatalities involving large trucks and buses.
Chairwoman Fischer, Ranking Member Booker, and Members of the  Subcommittee, this concludes my prepared remarks. I would be pleased  to answer any questions you or other Members may have at this time.
section_title: GAO Contact and Staff Acknowledgements
paragraphs: For further information regarding this statement, please contact Susan  Fleming at (202) 512-2834 or Flemings@gao.gov about this statement.  Contact points for our Offices of Congressional Relations and Public  Relations can be found on the last page of this statement. Matt Cook, Jen  DuBord, Sarah Farkas, Brandon Haller, Matt LaTour, and Amy  Rosewarne made key contributions to this statement.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.