Federal Motor Carrier Safety: Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers
Released: 2014-02-03T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.

GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes.

section_title: What GAO Found
paragraphs: The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers.

A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.

FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.

In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.

section_title: What GAO Recommends
paragraphs: GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations.

section_title: Letter
paragraphs: Large commercial trucks and buses are vital for the movement of goods  and people across America. According to the American Trucking  Associations, the trucking industry moved 9.4 billion tons of freight in  2012, and according to the American Bus Association, the “motor-coach”  industry provided about 694 million passenger trips in 2010. However,  this activity comes with a cost. From 2009 to 2012, crashes involving  large commercial trucks and buses averaged around 125,000 per year,  resulting in about 78,000 injuries and about 4,100 fatalities.

The primary mission of the U.S. Department of Transportation’s (USDOT)  Federal Motor Carrier Safety Administration (FMCSA) is to reduce  crashes, injuries, and fatalities involving large trucks and buses. FMCSA  partners with states to conduct roadside inspections and uses inspection  or crash information to assess and prioritize the riskiest motor carriers for  further intervention. From 1997 through 2010, FMCSA used a program  known as SafeStat to track how well motor carriers—the companies that  own commercial trucks and buses—complied with safety standards.  Under SafeStat, FMCSA reviewed only a small percentage of the more  than 500,000 motor carriers operating in the United States in a given  year. In an attempt to increase the number of motor carriers that FMCSA  can evaluate each year and, ultimately, to improve large commercial truck  and bus safety, FMCSA began to develop the Compliance, Safety,  Accountability (CSA) program in 2004. One component of the CSA  program is the Safety Measurement System (SMS), a data-driven  approach for identifying motor carriers at risk of presenting a safety  hazard or causing a crash. SMS uses information collected during  roadside inspections and from reported crashes to calculate scores  across seven categories that quantify a carrier’s safety performance  relative to other carriers.

Since 2008, when CSA was first piloted, law enforcement and industry  stakeholders have been generally supportive of FMCSA’s overall CSA  approach. Nonetheless, several evaluations of CSA conducted by a  range of outside groups concluded that some SMS safety scores  inaccurately assess a carrier’s relative crash risk. The precision and  accuracy of these scores is vital because FMCSA investigators and their  state partners use SMS results to focus their resources to help reduce the  number of motor carrier crashes, injuries, and fatalities. In addition,  FMCSA currently posts most of the scores publicly on its website for use  by industry stakeholders and the public and has indicated that a future  rulemaking will include similar information to help determine whether a  carrier is fit to operate motor vehicles.

We were directed in a Senate Appropriations Committee report to  continue monitoring FMCSA’s implementation of the CSA program.report examines the effectiveness of the CSA program in assessing  safety risk for motor carriers.

FMCSA provided us historical carrier data for several time periods, including December  2008, December 2010, June 2012, and December 2012. with varying levels of carrier exposure—measured by FMCSA as either  inspections or an adjusted number of vehicles. We assessed changes in  FMCSA’s requirements for carriers to receive SMS scores, changes in  SMS score calculation, and adjustments to the scoring weights. We also  evaluated the potential of FMCSA’s general approach to predict future  crashes by using data on violations of FMCSA regulations and crashes to  examine the relationships, if any, between violations of specific  regulations and subsequent crashes. Due to ongoing litigation related to  CSA and the publication of SMS scores, we did not assess the potential  effects or tradeoffs resulting from the display or any public use of these  scores.

Our analysis included nearly 315,000 U.S.-based carriers that were under  FMCSA’s jurisdiction and, with reasonable certainty, were active during  the period from December 2007 through June 2011. We considered a  carrier active during this period if it received a state or federal inspection,  was involved in a crash, or reported the number of vehicles it operates to  FMCSA. Information on inspections, violations, and crashes from  December 2007 through December 2009, our observation period, was  used to calculate SMS scores. We used crash information from the  remaining 18 month period—from December 2009 through June 2011— referred to as our evaluation period, to determine these carriers’  subsequent crash rates and involvement in crashes. Carriers in our analysis population accounted for approximately 120,000 reported  crashes during this 18-month period. Throughout this report, our analysis  is based on this population, during this time frame, unless otherwise  specified.

To identify any modifications to FMCSA’s method that could improve  effectiveness, we compared the results from our changes to FMCSA’s  existing methodology and identified an illustrative combination of changes  that better distinguished between carriers that later crashed and those  that did not. These illustrative changes included a change to the data  sufficiency standards for a carrier to receive an SMS score and changes  to the calculation method.

We also spoke with 1) FMCSA officials in its headquarters office, Western  Service Center in Colorado, and Colorado Division Office about the  implementation of CSA and 2) representatives from the Colorado State  Patrol and industry and safety interest groups. We selected Colorado  because it was one of the initial pilot states for CSA and has been  implementing the program since early 2008. We reviewed existing studies  and literature on CSA and Congressional testimony from industry and  safety interest representatives from a September 2012 hearing for the  House Transportation and Infrastructure Committee. Appendix I contains  a more detailed explanation of our scope and methodology. Appendix II  contains details about estimating rates of regulatory violations in the SMS  component of CSA. Appendix III contains details about the statistical  validity of the SMS component of CSA. Appendix IV describes prior  evaluations of SMS scores as measures of safety. Appendix V describes  our analysis of regulatory violations and crash risk. Appendix VI describes  the carriers we analyzed and provides the results from our analysis of  FMCSA’s methodology and our illustrative alternative.

We conducted this performance audit from August 2012 through February  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on the audit objectives.
subsections: section_title: Background
subsections: section_title: Motor Carrier Industry Diversity
paragraphs: The commercial motor carrier industry represents a range of businesses,  including private and for-hire freight transportation, passenger carriers,  and specialized transporters of hazardous materials. As of 2012, FMCSA  estimates that there were more than 531,000 active motor carriers, a  number that fluctuates over time due to the approximately 75,000 new  applications that enter the industry each year combined with thousands of  carriers annually leaving the market. Among carriers we assessed for this  report, most that operate in the United States are small firms; 93 percent  of carriers own or operate 20 or fewer motor vehicles. Nonetheless, a  large percentage of vehicles on the road are operated by large carriers.  Approximately 270 carriers have more than 1,000 vehicles each and  account for about 29 percent of all vehicles that FMCSA oversees.

section_title: FMCSA’s Role
paragraphs: FMCSA is responsible for overseeing this large and diverse industry.  FMCSA establishes safety standards for interstate motor carriers as well  as intrastate hazardous material carriers operating in the United States.  To enforce compliance with these standards, FMCSA partners with state  agencies to perform roadside inspections of vehicles and investigations of  carriers. In fiscal year 2012, FMCSA had a budget of approximately  $550 million and more than 1,000 FMCSA staff members located at  headquarters, four regional service centers, and 52 division offices.

In 2008, FMCSA launched an operational model test of CSA in four states  and began implementing the CSA program nationwide in 2010. CSA is intended to improve safety beyond the prior SafeStat program by  identifying safety deficiencies through better use of roadside inspection  data, assessing the safety fitness of more motor carriers and drivers,  and using less resource-intensive interventions to improve investigative  and enforcement actions. From fiscal year 2007 through fiscal year 2013,  FMCSA obligated $59 million to its CSA program, including CSA  development and technical support, information technology upgrades,  and training. For fiscal year 2014, FMCSA requested $7.5 million for  CSA.

CSA has three main components:

Safety Measurement System. SMS uses data obtained from federal or  state roadside inspections and from crash investigations to identify the  highest risk carriers. SMS was designed to improve on SafeStat by  incorporating all of the safety-related violations recorded during  roadside inspections. Carriers potentially receive an SMS score in  seven categories based on this information.  Intervention. A set of enforcement tools, such as warning letters,  additional investigations, or fines are used to encourage the highest  risk carriers to correct safety deficiencies, or place carriers out-of-  service.

Safety Fitness Determination Rule. This future rulemaking will amend  regulations to allow a determination—based in part on some of the  same information used to calculate SMS—as to whether a motor  carrier is fit to operate on the nation’s roads.

section_title: SMS Carrier Performance
paragraphs: SMS, the measurement system component of CSA, uses the data  collected from roadside inspections and crash reports to quantify a  carrier’s safety performance relative to other carriers. Specific carrier  violations recorded during roadside inspections are assigned to one of six  Behavioral Analysis and Safety Improvement Categories (BASIC).

According to FMCSA, these BASICs were developed under the premise  that motor carrier crashes can be traced to the behavior of motor carriers  and their drivers. A seventh category, called the Crash Indicator,  measures a carrier’s crash involvement history (see table 1). Each SMS  score is designed to be a quantitative determination of a carrier’s safety  performance.

For each of the approximately 800 violations that fall under the various  BASICs, FMCSA assigns a severity weight that is meant to reflect the  violation’s association with crash occurrence and crash consequence  when compared with other violations within the same BASIC. For  example, reckless driving violations, categorized in the Unsafe Driving  BASIC, are assigned a severity weight of 10 out of a possible 10 because  FMCSA determined that these violations have a stronger relationship to  safety risk than some other types of violations. Unlawfully parking, by  comparison, is also categorized in the Unsafe Driving BASIC, but is  assigned a severity weight of 1 out of 10.

FMCSA calculates SMS scores for carriers every month through a  process that has three main steps, each of which is made up of several  calculations.

Relevant inspections are either a driver inspection, in which the inspection focuses on  driver-related requirements, such as the driver’s record of duty or medical certificate, or a  vehicle inspection, which focuses on the condition of the motor vehicle. Driver inspections  are the relevant inspection for the Unsafe Driving, Hours-of-Service Compliance, Driver  Fitness, and Controlled Substances and Alcohol BASICs. Vehicle inspections are  considered relevant inspections for the Vehicle Maintenance BASIC. For the Hazardous  Materials BASIC, carriers that transport placardable quantities of hazardous materials are  also subject to vehicle inspections as the relevant inspections. Throughout the report, we  will refer to relevant inspections as simply inspections. another calculation—the number of vehicles a carrier operates  adjusted by the number of vehicle miles.

FMCSA accounts for exposure in order to make the scores comparable  across carriers. This approach has tradeoffs; while carriers can be  compared without penalizing some for having had more inspections or  road activity, exposure itself can be considered an element of risk. All  else being equal, carriers with more road activity are involved in more  crashes and potentially pose more risk to safety.

Step 2: Data sufficiency. Depending on the BASIC, carriers generally  receive SMS scores if they meet minimum thresholds of exposure (i.e.,  number of vehicles or inspections), or a minimum number of inspections  with violations (i.e., “critical mass”). For purposes of display on  FMCSA’s public website and identifying the highest risk carriers for  directing enforcement resources, FMCSA does not include scores for  carriers that do not meet a so-called critical mass of violations. For each  BASIC, this typically requires a minimum number of inspections that  include violations in that BASIC, a violation in that BASIC in the last 12  months, and, for some BASICs, a violation during the most recent  inspection.

Step 3: Dividing carriers into peer groups. After calculating violation  rates, FMCSA assigns carriers it determines have sufficient exposure to  peer groups with similar levels of on-road activity, or what the agency  refers to as safety event groups. According to FMCSA, safety event  groups are designed to account for the inherent greater variability in  violation rates based on limited levels of exposure and the stronger level  of confidence in violation rates based on carriers with higher exposure.  FMCSA assigns carriers to safety event groups based on their number of  inspections, the number of inspections with violations, or crashes the  carriers have accrued in the previous 2 years. Within each safety event  group, FMCSA calculates SMS scores by ranking carriers’ violation rates  (obtained in step 1 above) and assigning each carrier a percentile score  ranging from 0 to 100, where 100 indicates the highest violation rate and  the highest estimated risk for future crashes. FMCSA displays scores for  five of the BASICs on its public website.

section_title: Interventions
paragraphs: Once SMS scores are calculated, FMCSA begins a Safety Evaluation that  uses SMS scores to identify carriers with safety performance problems  requiring intervention. FMCSA has defined a fixed percentage threshold  for each BASIC that identifies those carriers that pose the greatest safety  risk. (For example, the threshold for the Unsafe Driving BASIC is 65 for  most carriers.) These carriers are then subject to one or more FMCSA  actions from a suite of intervention tools that were expanded as part of  CSA. Tools such as warning letters and on- and off-site investigations  allow FMCSA and state investigators to focus on specific safety  behaviors. FMCSA can also use enforcement strategies such as fines or  placing a carrier out-of-service. The range of available enforcement  options gives FMCSA investigators flexibility to apply interventions  commensurate with a carrier’s safety performance (see table 2). Seven of  the nine interventions are currently implemented nationwide. Prior to  CSA, FMCSA investigators’ only tool was a labor intensive,  comprehensive on-site investigation. With the additional set of  interventions, FMCSA aims to reach more carriers with its existing  resources.

According to FMCSA and state safety officials, an investigation or other  intervention can also be initiated based on the results of a crash  investigation, a complaint against a carrier, or a consistent pattern of  unsafe behavior by a carrier. FMCSA further designates some carriers  that exceed multiple BASIC thresholds as “high risk.” According to  FMCSA, many of these carriers are assigned a Safety Investigator, who  must complete a comprehensive review within a year regardless of any  changes in the carrier’s score. A carrier is considered high risk if it either: has an SMS score of 85 or higher in the Unsafe Driving BASIC or  Hours-of-Service Compliance BASIC or the Crash Indicator, and one  other BASIC at or above the intervention threshold, or exceeds the intervention threshold for any four or more BASICs.

section_title: Carrier Fitness to Operate
paragraphs: Currently, FMCSA can only declare a carrier as unfit to operate upon a  final unsatisfactory rating following an on-site inspection. In addition,  FMCSA can order a carrier to cease interstate operations if it determines  that the carrier is an imminent hazard. FMCSA can make this  determination for several reasons including:

FMCSA determining the carrier to be an imminent hazard.  receiving an “unsatisfactory” safety rating during an on-site  comprehensive investigation and failing to improve the rating within 45  or 60 days;  failing to pay a fine after 90 days;  failing to meet the standards required for a New Entrant Audit; or  According to FMCSA, during fiscal year 2012, the agency issued 855 out- of-service orders due to an unsatisfactory rating, 1,557 for failing to pay a  fine, and 47 because a carrier was determined to be an imminent hazard.

FMCSA has indicated its plans to propose using the same performance  data that inform SMS scores to determine whether a carrier is fit to  continue to operate. According to FMCSA, the Safety Fitness  Determination rulemaking would seek to allow FMCSA to determine if a  motor carrier is not fit to operate based on a carrier’s performance in five  of the BASICs, an investigation, or a combination of roadside and  investigative information. FMCSA proposes doing this through a public  rulemaking process; it currently estimates that it will issue a proposed rule  in May 2014.

section_title: CSA Program Increases Carrier Interventions, but FMCSA Faces Challenges in Identifying High Risk Carriers
paragraphs: CSA has been successful in raising the profile of safety in the motor  carrier industry and providing FMCSA with more tools to increase  interventions with carriers. However, FMCSA faces two major challenges  in reliably assessing safety risk for the majority of carriers in the industry  and prioritizing the riskiest carriers for intervention. First, we found that  the majority of regulations used to calculate SMS scores are not violated  often enough to strongly associate them with crash risk for individual  carriers. Second, for most carriers, FMCSA lacks sufficient safety  performance information to ensure that FMCSA can reliably compare  them with other carriers. FMCSA mitigates this issue by—among other  things—establishing data sufficiency standards. However, we found that  these standards are set too low, and by strengthening data sufficiency  standards SMS would better identify risky carriers and better prioritize  intervention resources to more effectively reduce crashes. Setting a data  sufficiency standard involves tradeoffs between scoring more carriers and  ensuring that the scores calculated are reliable for the purposes for which  they are used.
subsections: section_title: CSA Expands FMCSA’s Reach and Raises the Profile of Safety in the Industry
paragraphs: CSA has helped FMCSA reach more carriers and provided benefits to a  range of stakeholders. Since CSA was implemented nationwide in 2010,  FMCSA has intervened with more carriers annually than under SafeStat.  From fiscal year 2007 to fiscal year 2012, FMCSA increased its number  of annual interventions from about 16,000 to about 44,000, largely by  sending warning letters to carriers deemed to be above the intervention  threshold in one or more BASICs (see table 3). FMCSA and state  partners also took advantage of new ways to investigate carriers, such as  off-site investigations and on-site focused investigations, to complete 23  percent more investigations in fiscal year 2012 compared to fiscal year  2007 when only compliance reviews were used.

In addition, CSA provides data for law enforcement and industry  stakeholders about the safety record of individual carriers. For example,  as part of the CSA program, FMCSA publicly provides historical individual  carrier data on inspections, violations, crashes, and investigations on its  website. According to law enforcement and industry stakeholders we  spoke with, CSA organizes violation information for law enforcement and  carrier data related to the BASICs help guide the work of state inspectors  during inspections.

Law enforcement officials and industry stakeholders generally supported  the structure of the CSA program. These stakeholders told us that CSA’s  greater reach and provision of data have helped raise the profile of safety  issues across the industry. According to industry stakeholders, carriers  are now more engaged and more frequently consulting with law  enforcement for safety briefings. In Colorado, law enforcement officials  told us that CSA has improved awareness and engagement within the  motor carrier industry there. A state industry representative told us that  CSA has improved safety because carriers are in a competitive business  and can feel pressure to improve safety scores to gain an advantage over  the competition.

section_title: Relationship between Violation of Most Regulations and Crash Risk Is Unclear
paragraphs: The relationship between violation of most regulations FMCSA included in  the SMS methodology and crash risk is unclear, potentially limiting the  effectiveness of SMS in identifying carriers that are likely to crash.  According to FMCSA, SMS was designed to improve on its previous  approach to identify unsafe motor carriers by incorporating into the  BASICS all of the safety-related violations recorded during roadside  inspections. For SMS to be effective in identifying carriers that crash, the  violation information that is used to calculate SMS scores should have a  relationship with crash risk. Carriers that violate a given regulation more  often should have a higher chance of a crash or a higher crash rate than  carriers that violate the regulation less often. However, we found that  FMCSA’s safety data do not allow for validations of whether many  regulatory violations are associated with higher crash risk for individual  carriers. Our analysis found that most of the regulations used in SMS  were violated too infrequently over a 2-year period to reliably assess  whether they were accurate predictors of an individual carrier’s likelihood  to crash in the future. We found that 593 of the approximately 750  regulations we examined were violated by less than one percent of  carriers. Of the remaining regulations with sufficient violation data, we  found 13 regulations for which violations consistently had some  association with crash risk in at least half the tests we performed, and  only two violations had sufficient data to consistently establish a  substantial and statistically reliable relationship with crash risk across all  of our tests. (For more information, see app. V.) FMCSA attempted to  compensate for the infrequency of violations by, among other things,  evaluating aggregate data to establish a broader relationship between a   However, evaluations completed by  group of violations and crash risk.outside groups have found weaker relationships between SMS scores  and the crash risk of individual carriers than FMCSA’s evaluations of  aggregate data (for more information, see app. IV). SMS is intended to  provide a safety measure for individual carriers, and FMCSA has not  demonstrated relationships between groups of violations and the risk that  an individual motor carrier will crash. Therefore, this approach of  aggregating data does not eliminate the limitations we identified.

section_title: Most Carriers Lack Sufficient Information to Reliably Compare Safety Performance across Carriers
paragraphs: Most carriers lack sufficient safety performance information to ensure that  FMCSA can reliably compare them with other carriers. As mentioned,  SMS is designed to compare violation rates across carriers for the  purposes of prioritizing intervention resources. These violation rates are  calculated by summing a carrier’s weighted violations relative to each  carrier’s exposure to committing violations, which for the majority of the  industry is very low. About two-thirds of carriers we evaluated operate  fewer than four vehicles and more than 93 percent operate fewer than 20  vehicles. Moreover, many of these carriers’ vehicles are inspected  infrequently. (See table 14 in app. VI) Generally, statisticians have shown  that estimations of any sort of rate—such as the violation rates that are  the basis for SMS scores—become more reliable when they are  calculated from more observations. In other words, as observations  increase, there is less variation and thus more confidence in the precision  of the estimated rate. Given that SMS calculates violation rates for  carriers having a very low exposure to violations, such as operating one  or two vehicles or subject to a few inspections, many of the SMS scores   Carriers with  based on these violation rates are likely to be imprecise.few inspections or vehicles will potentially have estimated violation rates  that are artificially high or low and thus not sufficiently precise for  comparison across carriers. Further, because SMS scores are calculated  by ranking carriers in relation to one another, imprecise rate estimates for  some carriers can cause other carriers’ SMS scores to be higher or lower  than they would be if they were ranked against only carriers with more  reliable violation rates. This creates the likelihood that many SMS scores  do not represent an accurate or precise safety assessment for a carrier.  As a result, there is less confidence that SMS scores are effectively  determining which carriers are riskier than others. (App. II provides a  more technical discussion of these issues.)

For the five SMS BASICs for which FMCSA uses relevant inspections as  a measure of exposure—Hours-of-Service Compliance, Driver Fitness,  Controlled Substances and Alcohol, Vehicle Maintenance, and  Hazardous Materials—estimated violation rates can change by a large  amount for carriers with few inspections even when the number of their  violations changes by a small amount. For example, for a carrier with 5  inspections, a single additional violation could increase that carrier’s  violation rate 20 times more than it would for a carrier with 100  inspections. This sensitivity can result in artificially high or low estimated  violation rates that are potentially imprecise for carriers with few  inspections. As an example, our analysis of FMCSA’s method shows that  among carriers for which we calculated a violation rate for the Hours-of- Service Compliance BASIC, violation rate estimates are more variable for  carriers with fewer inspections. As shown in figure 1, violation rates tend  to vary by a larger amount across carriers with few inspections than  across carriers with more inspections. As a consequence, a high  estimated violation rate for a carrier with few inspections may reflect  greater safety risk, an imprecise estimate, or both. Further, comparisons  among carriers are meaningful only to the extent they involve carriers with  sufficient inspections and thus more precise estimated violation rates.

Similar to carriers with few inspections, carriers with few vehicles are also  subject to potentially large changes in their estimated violation rates,  which can affect a carrier’s SMS scores. For the Unsafe Driving BASIC  and the Crash Indicator, FMCSA measures exposure using a hybrid  approach that considers a carrier’s number of vehicles and its vehicle  miles traveled—when the latter information is available. Figure 2 shows  that among carriers for which we calculated a violation rate using  FMCSA’s method for the Unsafe Driving BASIC, carriers that operate  fewer vehicles, for example fewer than 5, experience a greater range in  violation rates per vehicle than carriers operating more vehicles, for  example, greater than 100. (For similar results on other BASICs, see  figures 10 to 16 in app. VI.)

Researchers have raised additional concerns about the quality and  accuracy of the data FMCSA uses to calculate SMS scores that could  potentially compound the problems with the precision of violation rate  estimates. These issues further limit the precision of carriers’ estimated  violation rates, and consequently their SMS scores. For example:

The frequency of an individual carrier’s inspections varies depending  on where the carrier operates. States vary on inspection and  enforcement practices. Some studies have shown that inspectors or  law enforcement officers in some states cite vehicles for certain  violations more frequently than in other states.

Delays in reporting crash data to FMCSA, as well as missing or  inaccurate data, can affect a carrier’s Crash Indicator SMS scores.  These delays can vary by state.

Data elements used to calculate violation rates for the Unsafe Driving  BASIC and Crash Indicator are based on information that is self  reported by the carrier. Inaccurate, missing, or misleading reports by a  carrier could directly influence their SMS scores. Additionally, among  carrier data we evaluated, more than 50 percent did not report their  vehicle miles traveled to FMCSA.

section_title: FMCSA Has Worked to Address Issues with Precision, but Its Methods Do Not Fully Address Limitations
paragraphs: FMCSA acknowledges that violation rates for carriers with low exposure  can be less precise and they attempt to address this limitation in two main  ways, but the methods incorporated do not solve the underlying  problems. As a result, SMS scores for these carriers are less reliable as  relative safety performance indicators, which may limit FMCSA’s ability to  more effectively prioritize carriers for intervention.
subsections: section_title: Data Sufficiency Standards
paragraphs: FMCSA established minimum data sufficiency standards to eliminate  carriers that lack what it has determined to be a minimum number of  inspections, inspections with violations, or crashes to produce a reliable  SMS score. For example, in the Hours-of-Service Compliance BASIC,  FMCSA does not calculate SMS scores for a carrier unless it has at least  three inspections and at least one violation within the preceding two  years. In addition, as previously mentioned FMCSA applies another data  sufficiency standard requiring a carrier to have a “critical mass” of  inspections with violations in order for an SMS score to be a basis for  potential intervention, or to be publicly displayed.

While this approach helps address the problems for carriers with low  exposure, it is not sufficient to ensure that SMS scores effectively  prioritize the riskiest carriers for intervention. For most BASICs, we found  FMCSA’s data sufficiency standards too low to ensure reliable  comparisons across carriers. In other words, many carriers’ violations  rates are based on an insufficient number of observations to be  comparable to other carriers in calculating an accurate safety score. Our  analysis shows that rate estimates generally become more precise  around 10 to 20 observations, higher than the numbers that FMCSA uses  for data sufficiency standards. However, the determination of the exact  data sufficiency standard needs to based on a quantitative measure of  confidence to fully consider how precise the scores need to be for the   (For more information, see  purposes for which the scores are used.app. II.)

section_title: Safety Event Groups
paragraphs: FMCSA groups the carriers meeting FMCSA’s data sufficiency standards  for each BASIC into safety event groups in order to, according to FMCSA,  “account for the inherent greater variability in violation rates based on  limited levels of exposure and the stronger level of confidence in violation  rates based on higher exposure.” based on inspections or inspections with violations depending on the  BASIC or on crashes for the Crash Indicator. For example, the first safety  event group in the Hours-of-Service Compliance BASIC includes carriers  that received from 3 to 10 inspections; the second group includes carriers  that received from 11 to 20 inspections, and so forth. Within each safety  event group, FMCSA rank orders carriers by violation rate and assigns a  percentile as an SMS score.

CSA, CSMS Methodology, Version 3.0.1, Revised August 2013. exceed FMCSA’s intervention thresholds at disproportionately higher  rates than carriers with more exposure. For example, FMCSA’s Hours-of- Service Compliance BASIC has five safety event groups. The group of  carriers with the fewest number of inspections in each safety event group  tends to have a higher percentage of carriers identified as above the  intervention threshold than the group of carriers with a greater number of  inspections (see fig. 3). This suggests that FMCSA’s methodology is not  adequately accounting for differences in exposure, as it is intended to do,  but rather is systematically assigning higher scores for carriers with fewer  inspections. (See figs. 17 to 25 in app. VI for other BASICs.)

FMCSA’s method of categorizing the carriers into safety event groups for  the remaining BASICs also demonstrates how imprecision  disproportionately affects small carriers. For the Unsafe Driving and  Controlled Substances BASICs, FMCSA forms safety event groups based  on the number of inspections with violations. Similarly, for the Crash  Indicator, safety event groups are based on a carriers’ number of  crashes. By using infractions or crashes to categorize carriers, FMCSA is  not addressing its stated intent of having safety event groups account for  differences in variability due to exposure. As a result, FMCSA derives  SMS scores for the Unsafe Driving BASIC and the Crash Indicator by  directly comparing small carriers with greater variability in their violation  rates—including many carriers with a violation rate based on one  vehicle—to larger carriers for which violations rates can be calculated  with greater confidence. We found that among carriers that received an  SMS score in Unsafe Driving, carriers with fewer than 20 vehicles are  more than 3 times as likely to be identified as above the intervention  threshold than carriers with 20 or more vehicles (see fig. 4). Of the  carriers operating one vehicle, nearly all were identified as above the  intervention threshold. (See figs. 26 to 32 in app. VI for other BASICs.)

FMCSA contends that these results are expected because only small  carriers that exceed critical mass standards receive an SMS score, and  small carriers that exceed this threshold have demonstrated several  occurrences of risky behavior despite their limited exposure. However,  this illustrates the volatility of rates and the disproportionate effect a single  violation can have given how FMCSA has structured SMS. For example,  using FMCSA’s data sufficiency standards, a carrier with one vehicle  (forty percent of the carriers in our analysis population have one vehicle)  and two inspections with unsafe driving violations does not have sufficient  information to be displayed or considered for intervention. However, a  single additional violation, regardless of the severity of the violation,  would likely mean that the carrier would be scored above threshold and  prioritized for intervention. A relatively small difference in the number of  violations could change a carrier’s status from “insufficient information”, to  “prioritized for intervention” with potentially no interim steps. Conversely,  a carrier such as this will have a very difficult time improving its SMS  score to be below threshold.

section_title: Strengthened Data Sufficiency Standards Can Improve FMCSA’s Ability to Identify High Risk Carriers
paragraphs: Our analysis shows that FMCSA could improve its ability to identify  carriers at higher risk of crashing by applying a more stringent data  sufficiency standard. As previously discussed, FMCSA uses SMS scores  to identify carriers with safety performance problems—those above the  threshold in any BASIC—for prioritization for intervention, and considers  carriers with SMS scores above the intervention threshold in multiple  BASICs as high risk. Overall, SMS is successful at identifying a group of  high risk carriers that have a higher group crash rate than the average  crash rate of all carriers that we evaluated. However, further analysis  shows that a majority of these high risk carriers did not crash at all,  meaning that a minority of carriers in this group were responsible for all  the crashes. As a result, FMCSA may devote significant intervention  resources to carriers that do not pose as great a safety risk as other  carriers, to which FMCSA could direct these resources. Given the issues  with precision discussed above, we developed and tested an alternative  to FMCSA’s method that sets a single data sufficiency standard, based  on the relevant measure of exposure—either at least 20 inspections or at  least 20 vehicles (depending on the BASIC), and eliminates the use of  safety event groups. This approach is designed to illustrate how a  stronger data sufficiency standard can affect the identification of higher  risk carriers and is not meant to be a prescriptive design to replace  current SMS methods.effect that including carriers with low levels of exposure and highly  variable violation rates can have on FMCSA’s prioritization of carriers for  intervention. Using this illustrative alternative, we found that FMCSA  would have more reliably identified a higher percentage of carriers that  actually had crashed than when compared to its existing methods. (Apps.  I and VI provide more detail on this approach.) Specifically:   The result of this analysis demonstrates the

This illustrative alternative identified about 6,000 carriers as high risk.

During the evaluation period of our analysis, these carriers’ group  crash rate was approximately the same as the rate for FMCSA’s high  risk group (about 8.3 crashes per 100 vehicles). However, a much  greater percentage of carriers (67%) identified as high risk using  alternative higher data sufficiency standards crashed, and these  carriers were associated with nearly twice as many crashes (see table  4).

For five out of six BASICs, the Crash Indicator, and the high-risk  designation, the illustrative alternative identified a higher percentage  of individual carriers above the intervention threshold that actually  crashed compared with FMCSA’s existing method. (See fig. 5.)

Using both FMCSA’s method and the illustrative alternative, for most  of the BASICs and the Crash Indicator the carriers identified above  the intervention threshold had a higher crash rate (crashes per 100  vehicles) than those below the intervention threshold (see table 5).  However, using FMCSA’s method, crash rates for the Controlled  Substances and Alcohol BASIC have the opposite, negative  association (3.2 crashes per 100 vehicles for carriers above threshold  versus 5.2 crashes per 100 vehicles for carriers below threshold),  whereas the illustrative alternative produces a positive association  (4.7 crashes per 100 vehicles for carriers above threshold versus 3.8  crashes per 100 vehicles for carriers below threshold).

Overall, these results raise concerns about the effectiveness of the  existing SMS as a tool to help FMCSA prioritize intervention resources to  most effectively reduce crashes. FMCSA’s existing SMS method  successfully identified as high risk more than 2,800 carriers whose  vehicles were involved in 12,624 crashes. However, FMCSA would have  potentially prioritized limited resources to investigate more than 4,000  carriers that did not crash at all. Prioritizing resources to these carriers  would limit FMCSA’s ability to reduce the number of overall crashes,  resulting in lost opportunities to intervene with the carriers associated with  many crashes.

Implementing a stronger data sufficiency standard as presented involves  tradeoffs between the number of carriers FMCSA can score, and the  reliability of those scores. Our analysis found that by increasing the data  sufficiency standards, fewer carriers would receive at least one SMS  score (approximately 44,000 carriers  in the illustrative alternative  versus approximately 89,000  using FMCSA’s method). The carriers  assigned an SMS score under the illustrative alternative accounted for  78.2 percent of all crashes during our evaluation period. FMCSA’s  existing method scores carriers responsible for about 85.9 percent of all  crashes (see table 6). On the other hand, by setting a higher standard for  data sufficiency, the illustrative alternative focuses on carriers that have a  higher level of road activity, or exposure, to more reliably calculate a rate  that tracks violations and crashes over the 2-year observation period. In  addition, exposure itself is a large determinant of overall risk, when  defined as a combination of threat and consequence, and could be used  as a factor to identify carriers that analysis suggest present a higher  future crash risk. This is consistent with the results in table 4 above,  which show that a larger proportion of the higher risk carriers in the  illustrative alternative crashed and were associated with a larger number  and proportion of crashes.

Regardless of where the data sufficiency standard is set, using only SMS  scores limits risk assessment for carriers that do not have sufficient  performance information. Our analysis shows that using FMCSA’s  existing method, about 28% of carriers have at least one SMS score,  leaving approximately 72% of carriers without any SMS scores—largely  due to insufficient information. The illustrative alternative scores fewer  carriers—14%, leaving 86% of carriers without any SMS scores.  However, according to an FMCSA official, there are other enforcement  mechanisms to assess and place unsafe carriers out-of-service, including  when a carrier fails to improve from an unsatisfactory safety rating during  a comprehensive review, fails to pay a fine, or FMCSA determines a  carrier is an imminent hazard. Further, the FMCSA official said carriers  that do not receive an SMS score can still be monitored because the  officials can initiate investigations and remove carriers based on  complaints and other initiatives. For example, FMCSA conducts  inspection strike forces targeting unsafe drivers and carriers in a  particular safety aspect, such as drug and alcohol safety records. These  tools used in conjunction with the performance data, including roadside  inspection and crash data, could provide FMCSA with complementary  means to assess and target carriers that do not otherwise have sufficient  data to reliably calculate SMS scores.

section_title: Precision Required in SMS Scores Depends on How They Are Used
paragraphs: The safety scores generated by SMS are used for many purposes, thus  the appropriate level of precision required depends on the nature of these  applications. According to FMCSA’s methodology, SMS is intended to  prioritize intervention resources, identify and monitor carrier safety  problems, and support the safety fitness determination process. In  setting a data sufficiency standard, FMCSA needs to consider how  precise the scores need to be, and a score’s required precision depends  on the purposes for which the scores are used.

FMCSA officials told us the primary purpose of SMS is to serve as a  general radar screen for prioritizing interventions. However, as discussed  above, due to insufficient data, SMS is not as effective as it could be for  this purpose. Further, if the same safety performance data used to inform  SMS scores are intended to help determine a carrier’s fitness to operate,  most of these same limitations will apply. According to FMCSA, the  Safety Fitness Determination rulemaking would seek to allow FMCSA to  determine if a motor carrier is not fit to operate based on a carrier’s  performance in five of the BASICs, an investigation, or a combination of  roadside and investigative information. FMCSA has postponed the planned rulemaking until May 2014. However, basing a carrier’s safety  fitness determination on limited performance data may misrepresent the  safety status of carriers, particularly those without sufficient data from  which to reliably draw such a conclusion.

In addition to using SMS for internal purposes, FMCSA has also stated  that SMS provides stakeholders with valuable safety information, which  can “empower motor carriers and other stakeholders…to make safety- based business decisions.” publicly released SMS scores stating that the data are intended for  agency and law enforcement purposes, and readers should not draw  safety conclusions about a carrier’s safety condition based on the SMS  score, but rather the carrier’s official safety rating. Nonetheless, entities  outside of FMCSA are also using SMS scores to assess and compare the  safety of carriers. For example:   FMCSA includes a disclaimer with the

The Department of Defense has written SMS scores into its minimum  safety criteria for selecting carriers of hazardous munitions.

FMCSA has released a mobile phone application—SaferBus—that is  designed to provide safety information, including SMS scores, for  consumers to use in selecting a bus company.

Multiple stakeholders have reported that entities such as insurers,  freight shippers and brokers, and others use SMS scores.

Given such uses, it is important that any information about SMS scoresmake clear to users, including FMCSA, the purpose of the scores, their  precision, and the context around how they are calculated. Stakeholders  have said that there is a lot of confusion in the industry about what the  SMS scores mean and that the public, unlike law enforcement, may not  understand the relative nature of the system and its limitations.

CSA, CSMS Methodology, Version 3.0.1 Motor Carrier Preview, Revised August 2013.

section_title: Conclusions
paragraphs: With the establishment of its CSA program, FMCSA has implemented a  data-driven approach to identify and intervene with the highest risk motor  carriers. CSA helps FMCSA to reach more carriers through interventions  and provides the agency, state safety authorities, and the industry with  valuable information regarding carriers’ performance on the road and  problems detected during roadside inspections.

GAO continues to believe a data-driven, risk-based approach holds  promise and can help FMCSA effectively identify carriers exhibiting  compliance or safety issues—such as violations or involvement in  crashes. However, assessing risk for a diverse population of motor  carriers—many of which are small and inspected infrequently—presents  several significant challenges for FMCSA. As a result, the precision and  confidence of many SMS scores is limited, a limitation that raises  questions about whether SMS is effectively identifying carriers at highest  risk for crashing in the future.

As presented in the report, strengthening data sufficiency standards is  one of several potential reforms that might improve the precision and  confidence of SMS scores. However, strengthening data sufficiency  standards involves a trade-off between assigning scores to more carriers  and ensuring that those scores are reliable. Our analysis shows how  improving the reliability of SMS scores by strengthening data sufficiency  standards could better account for limitations in available safety  performance information and help FMCSA better focus intervention  resources where they can have the greatest impact on reducing crashes.  In addition, if these same safety performance data are going to be used to  determine whether a carrier is fit to operate, FMCSA needs to consider  and address all identified data limitations, or these determinations will  also be at risk.

section_title: Recommendations for Executive Action
paragraphs: To improve the CSA program, the Secretary of Transportation should  direct the FMCSA Administrator to take the following two actions:  Revise the SMS methodology to better account for limitations in drawing  comparisons of safety performance information across carriers; in doing  so, conduct a formal analysis that specifically identifies:  limitations in the data used to calculate SMS scores including  variability in the carrier population and the quality and quantity of data  available for carrier safety performance assessments, and  limitations in the resulting SMS scores including their precision,  confidence, and reliability for the purposes for which they are used.

Ensure that any determination of a carrier’s fitness to operate properly  accounts for limitations we have identified regarding safety performance  information.

section_title: Agency Comments
paragraphs: We provided a draft of this report to the USDOT for review and comment.  USDOT agreed to consider our recommendations, but expressed what it  described as significant and substantive disagreements with some  aspects of our analysis and conclusions. USDOT’s concerns were  discussed during a meeting on January 8, 2014, with senior USDOT  officials, including the FMCSA Administrator. Following this meeting, we  made several clarifications in our report. In particular, FMCSA understood  our draft recommendation to be calling for specific changes to its SMS  methodology. It was not our intent to be prescriptive, so we revised our  first recommendation to state that FMCSA should conduct a formal  analysis to inform potential changes to the SMS methodology. In addition,  we clarified in the analysis and conclusions our meaning of reliability in  context of the purpose for which SMS is used.

We are sending copies of this report to relevant congressional  committees and the Secretary of Transportation. In addition, the report is  available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VII.

section_title: Appendix I: Scope and Methodology
paragraphs: This report addresses the effectiveness of the Compliance, Safety,  Accountability (CSA) program in assessing safety risk for motor carriers.  To assess how effectively CSA assesses the safety risk of motor carriers,  we reconstructed the models the Federal Motor Carrier Safety  Administration (FMCSA) uses to compute the SMS scores for all six  Behavior Analysis and Safety Improvement Categories (BASICs) and the  crash indicator. We then assessed the effect of changes to key  assumptions made by the models. Using data collected by the U.S.  Department of Transportation’s Motor Carrier Management Information  System (MCMIS) and historical SMS scores, and referencing the SMS  algorithm and methodological documentation, we replicated the algorithm  for calculating the SMS BASIC scores for the SMS 3.0 methodology.  Reconstructing FMCSA’s models and replicating the SMS scores FMCSA  produced for carriers was a necessary step to ensure that we understood  the complexities of the models, the data used in the calculation of the  SMS scores, and that the results we present in this report are comparable  to FMCSA’s outcomes. To corroborate our models with FMCSA’s, we  compared the SMS violation rates (measure scores) to FMCSA’s results  for December 2012. We assessed the reliability of data used, for our  purposes, by reviewing documentation on FMCSA’s data collection efforts  and quality assurance processes, talking with FMCSA and Volpe National  Transportation Systems Center officials about these data, and checking  the data for completeness and reasonableness. We determined that the  data were sufficiently reliable for the purpose of our data analysis.

We established a population of about 315,000 carriers for analysis that  were under FMCSA’s jurisdiction and showed indicators of activity over a  3 and a half year analysis period from December 2007 through June  2011. The criteria used to identify these carriers were:

U.S.-based carriers;  interstate or intrastate hazardous materials carriers;  carriers with at least one inspection or crash during the 2-year  analysis observation period (December 18, 2007 to December 17,  2009); and  carriers with a positive average number of vehicle count at any point  during the analysis observation period (December 18, 2007, to  December 17, 2009) and at any point during the evaluation period  (December 17, 2009, to June 17, 2011).

During the first 2 years of this period, December 2007 through December  2009, we used each carrier’s inspection, crash, and violation history to  calculate SMS scores. This period is referred to as the observation  period. The remaining 18 months, December 2009 through June 2011,  were classified as the evaluation period. We used data from this period to  identify carriers involved in a crash and estimate crash rates for these  carriers. For the approximately 315,000 carriers in our analysis, there  were approximately 120,000 crashes during the evaluation period. We  chose the lengths of time for observation and evaluation, in part, to match  FMCSA’s effectiveness testing methods.

We tested the effectiveness of SMS by identifying and making changes to  key assumptions of the model. Given FMCSA’s use of these scores as  quantitative determinations of a carrier’s safety performance, we  assessed the reliability of SMS scores as defined by the precision,  accuracy, and confidence of these scores when calculated for carriers  with varying levels of carrier exposure—measured by FMCSA as either  inspections or an adjusted number of vehicles. We tested changes to the  following characteristics of the model: the SMS measures of exposure,  the method used to calculate time weights, the organization of the  violations to the six BASICs, and the data sufficiency standards. To  evaluate the results produced by each model, including FMCSA’s, we  examined the SMS scores and classifications of carriers into the high risk  group. We compared the results from our revised models to the results  from a baseline model, SMS 3.0. For each model, we measured whether  carriers were involved in a crash, calculated group crash rates, and  calculated total crashes in the evaluation period for carriers that were and  were not classified as high risk in the observation period. Due to ongoing  litigation related to CSA and the publication of SMS scores, we did not  assess the potential effects or tradeoffs resulting from any public use of  these scores.

To determine the extent to which CSA identifies and intervenes with the  highest risk carriers, we examined how our changes to FMCSA’s key  assumptions affected the safety scores and identification of high risk  carriers. Specifically, we identified the carriers with SMS scores above  FMCSA’s intervention threshold in each BASIC and the carriers  considered high risk according to FMCSA’s high risk criteria. Using this  analysis, we designed an illustrative alternative method that incorporates  the following changes:  including only carriers with at least 20 observations in the following  measures of exposure: driver inspections when calculating scores for the Hours-of- Service Compliance, Driver Fitness, and Controlled Substances  BASICs;  vehicle related inspections for the Vehicle Maintenance BASIC;  vehicle related inspections where placardable quantities of  hazardous materials are being transported for Hazardous  Materials BASIC; and average power units for the Unsafe Driving and Crash Indicator assigning an SMS score to any carrier meeting these data sufficiency  standards (e.g., 20 inspections), even if that carrier does not have any  violations, was free of violations for 12 months, or had a clean last  inspection; eliminating safety event groups because of the stricter data sufficiency using only the average number of vehicles as the measure of  exposure for carrier’s assessed in the Unsafe Driving and Crash  Indicator BASICs.

Appendix VI provides the complete results of our replication of FMCSA’s  existing SMS and our illustrative revision to it.

We also examined the extent to which the regulatory violations that  largely determine SMS scores can predict future crashes. We developed  eight model groups to test the relationship between violations and  violation rates, and crashes. We tested only the violations that had non- zero variance and observations for at least 1 percent of the test  population. To control for small exposure measures when estimating  rates, we estimated models comparing carriers’ observed crash status to  Bayesian crash rates; used observed violation rates versus Bayesian  violation rates; and compared a full model sample to a restricted model  sample of carriers with at least 20 vehicles.sensitivity analysis to validate the predictive power of the models we  developed. We ran multiple variations of these models to determine the  number and types of violations that were predictive versus unstable. For   We also conducted a  more information on this specific analysis and model results, please see  appendix V.

In addition, we spoke with FMCSA officials in Washington, D.C., and at  the Western Service Center and the Colorado Division Office in  Lakewood, Colorado, and reviewed existing studies and stakeholder  concerns about the SMS model and its outcomes. To understand the  impact of CSA on law enforcement, we spoke with law enforcement  officials at the Colorado State Patrol. We selected Colorado because it  was one of the initial pilot states for CSA, and has been implementing the  program since early 2008. We also interviewed representatives from  industry and safety interest groups from the Colorado Motor Carriers  Association, the Commercial Vehicle Safety Alliance, and the American  Trucking Associations. Additionally, we attended meetings of the Motor  Carrier Safety Advisory Committee’s CSA subcommittee and reviewed  the minutes and related documentation from other meetings we did not  attend. We also reviewed congressional testimony from industry and  safety interest representatives from a September 2012 hearing for the  House Transportation and Infrastructure Committee. We reviewed  stakeholder comments submitted between March 2012 and July 2012 in  response to FMCSA’s planned improvements to SMS.

We conducted this performance audit from August 2012 to February 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Estimating Rates of Regulatory Violations in the Safety Measurement System
paragraphs: The FMCSA Safety Measurement System (SMS) methodology involves  the calculation of weighted violation rates for regulations within each of  six Behavioral Analysis and Safety Improvement Categories (BASICs)  and a given time period. (A seventh indicator measures weighted crash  rates in previous time periods, or “crash history.”) Carriers are assigned to  Safety Event Groups based on measures of their exposure to committing  violations, such as the number of driver or vehicle inspections, depending  on the BASIC, and the weighted violation rates are transformed into  percentiles for carriers within the same group. These percentiles  ultimately determine carriers’ alert or high-risk statuses. Because  regulatory violation rates strongly influence SMS scores, the precision  with which these rates can be calculated becomes important for  developing reliable measures of safety, as we discuss in the body of this  report.

In this appendix, we summarize statistical methods for estimating rates  and assessing their precision, or sampling error. We use these methods  to estimate crash rates and their sampling error for a population of motor  carriers that were active from December 2007 through December 2009.  Carriers may vary widely in their level of activity, known as “exposure.”  Both statistical theory and our analysis show that the precision of  estimated rates for carriers with low exposure, measured by vehicles or  inspections, is lower than for carriers with more exposure, and that rate  estimates can become distorted to artificially low or high values for these  low-exposure carriers. These results support our findings in the body of  this report on the precision of FMCSA’s current approach to calculating  safety risk scores and setting data sufficiency standards.
subsections: section_title: Statistical Methods for Estimating Violation Rates and Their Sampling Variance
paragraphs: Estimating rates of regulatory violations requires data on the number of  violations that carriers incur within a given time period. If one makes the  assumption that the number of violations is proportional to some measure  of exposure (activity) and also assumes that the probability of observing  violations within a large number of small independent exposure periods is  small, the sampling error of a rate estimate decreases as exposure  increases.

Specifically, assume that each carrier in a population of interest has a  unique violation rate, λ. For a fixed time period and known exposure, t,  the number of violations, V, is distributed as V ~ Poisson (λ t), with E(V)  = Var(V) = λ t. Since λ is unknown, it must be estimated from data on  regulatory violations and exposure.

The maximum likelihood (ML) estimator for a single carrier’s λ, given the  model above, is λı� = v / t, with Var(λı�) = λı� / t = v / t.the rate estimate increases exponentially as exposure decreases.  Accordingly, an estimated rate for a specific carrier and time period can  vary substantially from λ, particularly when exposure is low.

SMS is primarily concerned with measuring how regulatory violation rates  vary over a population of active motor carriers. Even though ordinary  methods of estimating these rates are unbiased and consistent, the  collection of estimated rates for the population, 𝛌� = {λestimates, such as the percentiles that SMS uses to place carriers into  alert and high-risk status, may be similarly prone to error.

Empirical Bayesian methods correct for this problem by estimating λı� for  each carrier to better estimate the distribution of rates across a  population. Bayesian methods prevent estimates from converging to  artificially extreme values for carriers whose raw rate estimates are based  on small samples (low exposure). The estimator does this by effectively  “borrowing information” from other, larger carriers whose rates can be  estimated more precisely. In the evaluation of the CSA Pilot Test for  FMCSA, the University of Michigan Transportation Research Institute  used empirical Bayesian rate estimation methods to evaluate the  association between SMS scores and crash risk, and cited similar  benefits to those we discuss here.

For example, see Roger J. Marshall, “Mapping Disease and Mortality Rates Using  Empirical Bayes Estimators,” Journal of the Royal Statistical Society, Series C (Applied  Statistics) 40, no. 2 (1991): 284, or J. N. K. Rao, Small Area Estimation (Hoboken, NJ,  2003), 206.

Specifically, assume that regulatory violation rates over a population of  carriers are distributed as λı� ~ Gamma(α, β), the prior distribution of the  parameter of interest. Parameter values for the prior distribution can be  assumed, based on historical data on the population of interest, or  estimated using a particular sample. Conditional on these rates, the data  on regulatory violations are distributed as V | λ , t ~ Poisson(λ t), and the  posterior distribution for a specific carrier is given by  λ | v, t ~ Gamma(α + v, β + t) (1)

Since the mean of a Gamma variate is α / β and the variance is α / β, the  posterior mean and variance of the rate for a given carrier are given by  E(λ | v , t) = (α + v) / (β + t) (2)

Var(λ | v , t) = (α + v) / (β + t), and the mean of the prior  distribution, α / β. When enough data are available, as indicated by a  large exposure term relative to the violation term, the estimate converges  to the ordinary, carrier-specific rate estimate. When exposure is low,  however, the method combines data from the specific carrier with the  mean rate for all carriers.

The variance of Bayesian rate estimates decreases with increased  exposure, similar to the variance of ordinary rate estimates. Figure 6  shows how hypothetical rate estimates and 90% posterior intervals for a  carrier that experienced 5 crashes vary with the carrier’s exposure, as  measured by the number of vehicles. (Although we illustrate rate  estimation issues using crash rates, we likely would have obtained similar  results if we had estimated regulatory violation rates.) As expected, the  precision of the estimates decreases exponentially as the number of  vehicles increases. The variance is high in the range of 1 to 5 vehicles  and begins to decrease less quickly at approximately 20 vehicles,  consistent with our discussion in the body of this report and prior  evaluations of SMS.

Thresholds in this approximate range are consistent with criteria used by  the Centers for Disease Control and Prevention (CDC) to suppress or  caveat rate estimates for the purpose of public display. For example, in  its compendium of health statistics in the United States, CDC cautions  that “hen the number of events is small and the probability of such an  event is small, considerable caution must be observed in interpreting the  conditions described by the figures.”

Even though the Bayesian estimates do not converge to extremely low or  high values when exposure is low, the uncertainty around the estimates  remains high. As figure 6 shows, statistical methods for modeling and  estimating rates can quantify this uncertainty explicitly, in order to reflect  the varying precision of estimates for motor carriers with more or less  observed data. Although the amount of uncertainty that is acceptable in  practice depends on the purpose of the estimates, both statistical theory  and government agencies estimating rates similar to those involved in the  calculation of SMS scores have recognized the need to express the  uncertainty of these estimates, particularly when the derived from small  samples. This contrasts with FMCSA’s approach, which reports SMS  scores as safety risk estimates with no quantitative measures of  precision.

section_title: Applying Rate Estimation Methods to Motor Carrier Data
paragraphs: To illustrate the rate estimation issues discussed above in the context of  motor carrier safety, we estimated individual crash rates for a population  of motor carriers that were actively operating in each of two time periods,  December 2007 through December 2009, and December 2009 through  June 2011, as measured in FMCSA’s Motor Carrier Management  Information System (MCMIS). An “active” carrier was one that, in each  time period, had at least one inspection or crash and had been recorded  as a US-based interstate or intrastate Hazmat carrier. This definition  resembled the one we used in replicating SMS, as described in the body  of this report and appendix I. We obtained these data from the December  2010 and December 2012 MCMIS “snapshot” data files, as well as a  historical file of carrier-specific information that covered all snapshots.

We estimated the raw and empirical Bayesian crash rates for each carrier  in the first time period, using data on the number of crashes and vehicles  for these carriers and the formulas above. We used the “empirical Bayes”  version of the rate estimator, in which the parameters of the prior  distribution were estimated from the data. Specifically, we fit the observed  rate data for all carriers in the first time period to the negative binomial  distribution, parameterized with exposure measured by number of  vehicles, and estimated α and β using standard methods of maximum  likelihood estimation. The final rate estimates for each carrier were a  combination of these parameter estimates and carrier-specific data,  according to equation 2 above.

As theory would predict, Bayesian methods prevented crash rates from  converging to zero or extremely high values for carriers with low  exposure. The left half of figure 7 presents the raw crash rates for our  analysis carriers, while the right half presents the empirical Bayesian  estimates. The raw estimates for carriers with about 1 to 10 vehicles can  be 10 to 20 times higher than for carriers with more than 10 vehicles. In  addition, the raw rates cluster at zero for a large number of carriers,  particularly for those with low exposure. An underlying crash rate of zero  is implausible for active carriers. In contrast, the Bayesian rate estimates  are more stable, with no inflation or deflation to extreme values. Since the  body of this report finds that 93 percent of carriers in our replication of  SMS had fewer than 20 vehicles, Bayesian methods may provide more  stable estimates for many specific carriers and may better approximate  the distribution of rates across carriers.

In addition to stabilizing rates for small carriers, Bayesian rate estimation  methods provide an explicit measure of precision for each carrier’s rate,  regardless of size. In figure 8, we show the Bayesian rate estimates for a  random sample of 109 carriers in the first period of our analysis  population, along with 90 percent Bayesian posterior intervals. (We  present these results for a sample to make the intervals readable.) The  posterior interval expresses the range over which the true rate exists with  a 90 percent probability. Consistent with theory, the precision of the rate  estimates increases with exposure—in this case, the number of vehicles.  These results apply to actual carriers in the sample, but the results are  consistent with those expected by theory. The width of the posterior  intervals does not decrease monotonically, however, because the relative  number of crashes also affects the variance and is not held constant in  the plot.

section_title: Appendix III: Evaluating the Statistical Validity of the Safety Measurement System
paragraphs: In this appendix, we express the Safety Management System (SMS) as a  statistical measurement model, in order to make its assumptions explicit,  and describe how estimating the model could validate those assumptions.  We find that FCMSA’s SMS makes a number of strong assumptions  about motor carrier safety that empirical data cannot easily validate.

The SMS uses administrative data on inspections of commercial motor  carriers, violations of regulations, and crashes to measure carrier safety.  Statisticians and other researchers have developed methods to validate  measures of such broad concepts as safety, referred to as “latent  variables,” using empirical data. These methods are known as  “measurement models.” For example, mental health professionals have  created scales to measure the existence of broad disorders, such as  depression, by combining responses to multiple items on patient  questionnaires. SMS has a similar goal: to create scales to measure  motor carrier safety risk on several dimensions, such as “Unsafe Driving”  or “Vehicle Maintenance,” by combining violation rate data across multiple  regulations. Latent variable measurement methods can assess whether  these broader measures are valid and reliable, and whether the empirical  indicators that go into them actually measure the intended concepts.  Estimating the degree to which various indicators measure a broader  concept helps confirm and often improve the reliability and validity of the  scales constructed.
subsections: section_title: Structure and Assumptions of SMS
paragraphs: Much of the SMS involves calculating weighted regulatory violation rates  for motor carriers in a given time period. FMCSA assigns weights that, in  principle, reflect the violations’ associations with one of six dimensions of  safety, known as Behavioral Analysis and Safety Improvement  Categories (BASICs), such as “Unsafe Driving” and “Vehicle  Maintenance.”

The weights represent what FMCSA considers to be the  strength of each violation’s association with safety, relative to other  violations in the same BASIC. All violations that are categorized in a  BASIC get a positive weight ranging from 1 to 10, which implies that they  have some association with safety. These weighted violation rates  strongly influence the final SMS measures of safety on these dimensions.  Each BASIC is linked to a set of violations, which are all assumed to  measure the same dimension of safety. Each violation maps to exactly  one BASIC, though BASICs map to multiple violations in their associated  groups. .

Vij measures the number of times that carrier i violated regulation j in a  given time period. 𝜆𝑗 is a weight for each violation. It is the product of a  “severity” weight, measuring what FMCSA considers the violation’s “crash  risk relative to the other violations comprising the BASIC measurement,”  in addition to outcomes thought to be particularly severe (e.g., out-of- service violations), and a time weight, measuring what FMCSA considers  the importance of violations from different time periods to estimating a  carrier’s current level of safety. By defining Vij for fixed time periods, such  as 6 or 12 months prior to the measurement time, we collapse the  separate weights used in SMS into 𝜆𝑗, in order to simplify the notation.

Lastly, T measures exposure to committing violations in the time period,  which is either a function of carrier’s vehicles and vehicle miles traveled  (VMT) or the time-weighted sum of relevant inspections, depending on  the BASIC.

SMS transforms the weighted violation rates for each carrier into  percentile ranks, after applying a number of “data sufficiency standards”  to exclude carriers with few violations, inspections, and/or vehicles.  Carriers with percentiles that exceed established thresholds are “alerted”  on the relevant BASICs and, if enough alerts or other conditions exist, are  identified as “high risk.” As a result, the ultimate measures of safety risk  are ordered groups, with cut-points defined by BASIC percentiles for  carriers that meet FMCSA’s standards for data sufficiency.

section_title: SMS as a Latent Variable Measurement Model
paragraphs: The SMS can be viewed as an attempt to measure latent concepts of  “safety,” such as “Unsafe Driving” or “Vehicle Maintenance,” using  observed data on regulatory violations and the opportunity to commit  them (exposure). Consider the latent variable measurement model below,  using notation from a prominent textbook:  The weights describing the relationship between the latent and observed  The model assumes that a vector of 𝑘=∑ 𝑛𝑔𝑝𝑔 observed variables, r, are  determined by p latent variables, 𝜉, and random measurement error, 𝛿.  variables make up the block diagonal matrix Λ, with p blocks of weights applied to the corresponding blocks of observed variables. This structure  latent variable. In many applications, the model assumes that Cov(𝜉 implies that each group of observed variables is related to exactly one  , 𝛿)=0 and E(𝛿 ) = 0 but allows other variances and covariances to be estimated from the data as parameters or fixed to known values.

The SMS is a particular form of the model above. Specifically, SMS  defines r as violation rates for k = 826 regulations, where r may include  variables measured at different times. It sets p = 6 and relates the  violation rates to the BASICs, or latent variables 𝜉 measuring safety,  through the weighting matrix Λ. FMCSA created fixed time and severity  assumes that 𝛿=0. A graphical version of SMS as a measurement   weights for each regulation through a combination of statistical analysis  and the opinions of stakeholders. Since SMS is not a stochastic model, it  model appears in figure 9 below.

When expressed as a measurement model, the strong assumptions of  SMS —and their potential detrimental effect on its usefulness—become  clear. FMCSA’s assumption of zero measurement error is unusual for  statistical approaches to measurement, given that any particular violation  is likely to represent variation in latent variables (in this case, safety) as  well as unmeasured variables summarized by the error term. SMS makes  specific assumptions about the number of safety dimensions—the latent  variables assumed by the model above—as well as their relationships to  violation rates. Exactly six dimensions of safety exist (involving  regulations), and each violation rate measures only one of them. In other  efforts to measure broad concepts using numerous indicators, inference  about the existence and relationships among observed and latent  variables are endogenous parameters (determined by the model) to be  estimated, rather than exogenous parameters (determined outside the  model) that are fixed ex ante, ahead of time, as they are here. Finally,  SMS takes the unusual step of fixing the values of the weights relating the  latent variables measuring safety to violation rates at values other than 0.  This assumes a high degree of prior knowledge about the relationships  between latent and observed variables. Although FMCSA has conducted  several studies of how regulatory violation rates are associated with crash  risk, these studies do not directly estimate the degree to which each type  of violation reflects one of several dimensions of safety.

One approach to validating the assumptions of SMS is to estimate the  parameters of the measurement model above using empirical data on  regulatory violation rates. This approach is known as Confirmatory Factor  Analysis, which is a special type of measurement model. Because SMS  makes specific assumptions about the number of BASICs and the  violations that go into them, we can express the system as a  measurement model, as discussed above, and estimate the degree to  which its assumptions are consistent with reality. For example, SMS  assumes that six dimensions of safety exist—labeled BASICs in SMS— and that each violation reflects only one dimension. However, a model  that assumes three BASICs and allows violations to reflect multiple  dimensions of safety might be a plausible alternative. High violation rates  for brake maintenance regulations may indicate worse performance on  both the Vehicle Maintenance and Unsafe Driving dimensions of safety.  Measurement modeling can identify which of these approaches better fits  empirical patterns of regulatory violations. More generally, analyzing SMS  as a measurement model can validate its assumptions, such as the  values of the severity and time weights, and suggest improvements to  better measure safety.

We can extend the SMS measurement model to predict empirical data on  crash risk, in order to further validate its ability to identify high-risk  carriers. This structural equation modeling (SEM) approach combines the  measurement model above with a model that describes how the latent  dimensions of safety predict crash risk, generically known as  “endogenous observed variables.”

To incorporate outcomes, we extend the measurement model above to  assume that the six BASICs are directly related to an empirical measure  of crash risk:  C measures crash risk; 𝛾 are parameters describing how the latent safety  dimensions are related to crash risk; 𝜉 are the safety dimensions; and 𝜀𝑖  parameters describing how the SMS scores relate to crash risk, 𝛾. Strong  is a random error term. Estimating this larger model would yield the  original parameters of the measurement model, in addition to the  correlations between SMS scores and crash risk would further support  their ability to identify higher-risk carriers. This is known as “criterion  validity” in statistics and social research.

A key strength of this validation approach is that it accounts for the error  in measuring broad dimensions of safety when predicting crash risk.  Because empirical data on violation rates and SMS scores are indicators  of latent concepts of safety, measurement error can distort the underlying  relationships between these broader concepts and crash risk. For  example, poor vehicle maintenance may be positively associated with  higher crash risk, but empirical data on violations of vehicle maintenance  regulations may measure both the concept of interest and the  enforcement efforts of state and local governments. As a result, the  violation rates may be uncorrelated with crash risk simply due to error in  measuring the concept of interest. SEM models estimate the relationships  among latent variables more precisely by accounting for this  measurement error. This contrasts with simpler regression models of  crash risk as a function of observed violation rates, which assume that  violation rates measure the dimensions of safety without error.

section_title: Appendix IV: Prior Evaluations of SMS Scores as Measures of Safety for Specific Carriers and Risk Groups
paragraphs: Previous evaluations of SMS have focused on estimating the correlations  between crash risk and regulatory violation rates and Safety  Measurement System (SMS) scores. These evaluations have found  mixed evidence that SMS scores predict crash risk with a high degree of  precision for specific carriers or groups of carriers. This appendix  synthesizes the results of these prior evaluations.

Several prior evaluations of SMS have analyzed grouped data, rather  than directly analyzing how a carrier’s individual regulatory violation rates  and SMS scores predict its own future crash risk. For example, in a pilot  evaluation conducted for FMCSA, the University of Michigan  Transportation Research Institute (UMTRI) estimated group crash rates  within percentiles of SMS scores for each Behavioral Analysis and Safety  Improvement Category (BASIC), pooling several hundred carriers in each  percentile, to trace out the aggregate relationship between SMS scores  and crash risk. Similarly, FMCSA’s Violation Severity Assessment Study  analyzed grouped violation data from roadside inspections conducted  from 2003 through 2006, in order to compare rates cited in post-crash  reports to rates in the general population of carriers.

Ibid., 4-2, 4-6. that did and did not exceed the SMS thresholds to be placed in “alert” or  “high risk” statuses.

Aggregate approaches, such as those used in several prior evaluations,  do not directly assess the ability of SMS and regulatory violations to  predict future crash risk for specific carriers. Well-known findings in  statistics on “ecological fallacies” show that associations at higher levels  of analysis are not guaranteed to exist at lower levels of analysis. In this  application, carriers that crash may have higher violation rates or SMS  scores as a group than carriers that do not crash, but this pattern does  not necessarily apply to specific carriers within the groups. Because less  variation exists at the carrier level, aggregation can overstate the strength  and precision of these correlations for individual carriers.

Even when similar correlations exist at the carrier level, comparing  average crash rates for SMS percentiles or risk groups does not assess  the prediction error for any particular carrier. The average crash rate may  be higher for groups of carriers with increasingly high SMS percentiles,  but crash rates may vary significantly around these means. This residual  variation, not differences in means or other aggregate statistics, is more  directly relevant for assessing the quality of predicted crash rates for a  particular carrier. In statistical terms, the prediction error summarized by  the residual variance of a linear regression model or the classification  matrix of a categorical model is what matters for assessing predictive  power for individual carriers, not the models’ coefficients, which estimate  mean crash rates conditional on these percentiles.

Thus, it is not surprising that previous evaluations of carrier-level data  have found weaker relationships between crash risk and SMS scores and  regulatory violations than have the evaluations of aggregated data.

UMTRI estimated the relationship between exceeding thresholds in the  six non-crash BASICs and mean crash rates, using an empirical Bayesian  negative binomial model estimated on carrier-level data. The results  showed that carriers exceeding the thresholds for the Unsafe Driving and  Vehicle Maintenance BASICs had average crash rates that were 1.1 to  1.8 times higher than carriers not exceeding the thresholds—usually  lower than the rate ratios of 1.0 to 5.4 reported by UMTRI’s aggregate  analysis and FMCSA’s December 2012 Effectiveness Testing. However,  this relationship was negative for the Driver Fitness and Loading/Cargo  (currently Hazardous Materials) BASICs, with mean crash rates for  alerted carriers that were 0.85 and 0.91 times the rates of non-alerted  carriers, respectively. The ratios were not significantly greater than 1 for  the Fatigued Driving and Substance Abuse/Alcohol BASICs. Similarly,  the American Transportation Research Institute (ATRI) found that alerted  carriers in the Unsafe Driving, Vehicle Maintenance, Hours-of-Service,  and Controlled Substances/Alcohol BASICs had mean crash rates that  were 1.3 to 1.7 times larger than scored carriers not in alert status, but  carriers exceeding the Driver Fitness thresholds had mean crash rates  that were 0.87 times those of non-alert scored carriers.

Although UMTRI and ATRI analyzed carrier-level data, they validated  SMS measures using regression coefficients and similar statistics that  describe aggregate correlations. As we discuss above, this approach  does not directly quantify predictive power for specific carriers.

Two studies that have directly estimated prediction error for specific  carriers, conducted by Wells Fargo Securities and James Gimpel of the  University of Maryland, found weaker evidence of the model’s predictive  effectiveness. Gimpel found that mean crash rates increased by small  amounts as SMS scores increased on the Unsafe Driving, Hours-of- Service, and Vehicle Maintenance BASICs increased.found a similarly positive association for the Unsafe Driving BASIC, but a   Wells Fargo  negative association for the Hours-of-Service BASIC, in its analysis of  4,600 carriers with at least 25 vehicles and 50 inspections. More  critically, the authors showed that scores on these BASICs predict crash  rates with a large amount of error, with most R-squared fit statistics  ranging from nearly zero to 0.07 for reasonably large analysis samples.  Although these studies do not report critical estimates of the residual  variance, the R-squared statistics likely imply confidence intervals around  predicted crash rates for individual carriers with widths that are several  times larger than the predictions themselves. This implies that SMS  scores predict future crash risk for specific carriers with substantial error,  even though mean crash rates can be higher among carriers with higher  SMS scores.

FMCSA used aggregate data to dispute the findings of the Wells Fargo  evaluation. Specifically, the agency cited the UMTRI findings that  aggregate crash rates were 3.0 to 3.6 times higher for carriers exceeding  thresholds for the Unsafe Driving and Hours-of-Service BASICs than for  carriers that did not exceed thresholds for any BASIC. In addition,  FMCSA highlighted analyses by UMTRI and the Volpe Center of  aggregate crash rates across percentiles of SMS scores in the Unsafe  and Fatigued Driving BASICs, respectively, which they claimed to show a  stronger correlation to crash risk. FMCSA’s approach to evaluating the  predictive power of SMS scores resembles its Effectiveness Testing,  which compares aggregate crash rates for carriers above and below  thresholds for various BASICs.

However, as we discuss above and Wells Fargo discussed in its  response to FMCSA, the fact that SMS scores predict aggregate crash  rates more strongly at the alert-group or percentile level does not  necessarily imply that the scores will predict the crash risk of individual  carriers. Recognizing this, the UMTRI evaluation analyzes the data at  both the aggregate and carrier levels, and finds that mean crash rate  ratios are far smaller at the carrier level than at the alert-group or  percentile levels. It should be intuitive that aggregate evidence of  effectiveness, stressed in some FMCSA evaluations, shows stronger  predictive power than the carrier-level analyses of ATRI, Gimpel, UMTRI,  and Wells Fargo. Aggregating violation and crash rates within larger  groups effectively increases the sample size used to calculate rates,  which reduces their sampling error when compared to the equivalent  carrier-level measures. The reduction of sampling error can strengthen  the correlations between violation rates and SMS scores and crash risk.

Evaluations of SMS that focus on carrier-level prediction error provide the  most appropriate evidence of effectiveness for assessing the safety of  individual carriers. FMCSA has stated that one purpose for SMS scores is  to predict the future crash risk of individual motor carriers, in order to  prioritize resources for intervention and enforcement. In addition, FMCSA  reports SMS scores as measures of safety on a public website and the  SaferBus Mobile app. To assess the validity of SMS scores for this  purpose, evaluations should focus on the system’s ability to predict the  crash risk at the carrier level, not its ability to identify groups of carriers  with larger crash rates on average or collectively. Measures of predictive  accuracy—such as the residual error made when predicting crash rates  or the classification error made when assigning carriers to risk groups— are the critical metrics of success, not aggregated crash rate ratios and  regression coefficients. When evaluated on these criteria, prior studies  show that SMS predicts future crash risk for individual carriers with  substantial imprecision.

None of the prior studies has explicitly incorporated measurement error  into evaluations of SMS. Since SMS is ultimately a method of creating  measures of latent variables, as we discuss in appendix III, the  regulations used to calculate scores and the scores themselves have  some degree of measurement error. Because existing studies have used  statistical methods that assume zero measurement error, more  comprehensive attempts to model the measurement structure of SMS  and validate its assumptions and predictive power, such as those we  discuss in appendix III, may produce different results. The correlations  among SMS scores, violation rates, and crash risk may reflect  measurement error as much as the underlying relationships among the  variables of interest. This more complex analysis is critical for future  evaluations of SMS and its ability to measure safety risk.

section_title: Appendix V: Analysis of Regulatory Violations and Crash Risk
paragraphs: As a more basic approach to validating SMS, which focuses on the ability  of data on regulatory violations in one time period to predict crash risk in a  subsequent period, we analyzed the relationship between violation rates  and crash risk using a series of statistical models. These models  predicted the probability of a crash and crash rates as a function of  regulatory violation rates for a population of motor carriers that were  actively operating over a recent 3.5-year time period (described below).

We find that a substantial portion of regulatory violations in SMS cannot  be empirically linked to crash risk for individual carriers. Consistent with  prior research, about 160 of the 754 regulations with data available in  this time period had sufficient variation across carriers for analysis. Of the  approximately 160 regulations with sufficient violation data, less than 14  were consistently associated with crash risk, across statistical models.  These results suggest that the specific weights that SMS assigns to many  regulations when calculating safety risk cannot be directly validated with  empirical data, and many of the remaining regulations do not have  meaningful associations with crash risk at the carrier level.
subsections: section_title: Data and Methods
paragraphs: We assembled data for a population of motor carriers using the MCMIS  snapshot files dated December 2010 and 2012. Specifically, we identified  carriers that were actively operating in each of two time periods: from  December 2007 through December 2009 (the “pre-period”) and from  December 2009 through June 2011 (the “post-period”). We defined an  active carrier as one that is as outlined in Appendix I, consistent with  FMCSA’s definition of active carriers for its Effectiveness Testing and  other analyses. For each of the approximately 315,000 carriers that met  these criteria, we extracted data on the number of regulatory violations  and crashes incurred in each time period, along with the number of  inspections, vehicles, and use of straight versus combo trucks, among  other variables, from the crash and inspection tables in MCMIS.

The goal of our analysis was to predict crash risk in the post-period, using  data on regulatory violations, crash data, and carrier characteristics  measured in the pre-period. We developed a series of linear and  generalized linear regression models to predict two measures of crash  risk for individual carriers: a binary indicator for having crashed in the  post-period and the ratio of crashes to vehicles. Estimating and  evaluating all potential models and model types was not the goal of these  analyses. Rather, we sought to estimate the associations between  regulatory violation rates and crash risk at the carrier level, in order to  validate the violations’ severity weights in SMS.

We reduced the list of 754 regulations whose violations are tracked in  MCMIS to those that had enough variation across carriers for analysis.  After excluding 593 violations that had zero variance or zero counts for  more than 99 percent of the analysis carriers, we retained data on the  violation of approximately 160 regulations for use in predicting crash risk.

As we discuss in appendix II and the body of this report, crash and  violation rates based on small exposure measures, generally resulting  from carriers with few vehicles, may be estimated with less precision than  rates based on larger exposure measures. To better understand and  attempt to overcome these rate estimation issues and assess the  sensitivity of our results, we used both ordinary and empirical Bayesian  estimators of crash and violation rates. In addition, we estimated  separate models limited to carriers that had more than 20 vehicles.

These methodological choices produced 8 groups of models, as  described in table 7. The groups were defined by the combined  categories of crash measure (binary crash status versus Bayesian crash  rate), methods of violation rate estimation (ordinary versus Bayesian),  and carrier size (full data or restricted to more than 20 vehicles). These  parallel analyses allowed us to assess the sensitivity of our results to  different assumptions.

For each of the eight model groups, we include three sets of covariates to  predict crash risk in the post-period:  “Simple model:” indicator (binary) for crashing in the pre-period,  carrier size, and carrier type (percent straight versus combo).  “Full model:” predictors in the simple model, plus all violation rates  with viable data in the pre-period.  “Stepwise full model:” We applied a stepwise selection algorithm  applied to all predictors in the “full model,” in order to select the most  predictive covariates. The algorithm’s constraints required a p-value of  0.30 for a covariate to enter the model and 0.35 to remain in the  model.

To avoid over-fitting our models to any particular sample of data, we  divided our data using a random method to form a model-building sample  and a validation sample. We used the model-building sample to estimate  the models described above and the validation sample to assess the  accuracy of the model’s predictions of crash probability against new data.  When seeking to develop statistical methods for predictive purposes, this  type of out-of-sample validation is extremely useful to ensure that any  method identified can consistently predict well on all samples of data, not  just the sample that was used to develop the method. This is an important  limitation of prior evaluations of SMS, which, to our knowledge, have not  used replication samples to avoid over-fitting when identifying predictive  violation types or methods of identifying higher-risk carriers.

Model selection required addressing statistical estimation issues, such as  instability of the parameter estimates caused by co-linearity of predictors  or lack of variability in the predictors, and other model fitting concerns.

For the linear crash rate models, the dependent variable required a log  transformation to remove non-constant error variance, which would  invalidate results if left untreated. These statistical issues resulted in sub- models within the major model groups that were explored until a stable  model resulted. Therefore, the results within each model group focus on  three sub models, when applicable: simple, stepwise and full, where  stepwise is the model that eliminated independent variables until a  stabilized model with estimable coefficients resulted. See table 8 for the  final list of 30 models and subsamples.

section_title: Evaluation of Models
paragraphs: Models that use the SMS violation information do not fit well according to  various measures discussed below. In addition, the violation rates, as  measured in SMS, do not have a strong predictive relationship with  crashes, regardless of whether the observed or the Bayesian violation  rates are used as inputs.

Models for crash status (yes/no) were examined for stability of parameter  estimates, fit statistics, number and types of violations that were  predictive and that were stable, and future predictive performance  according to these measures. Models for Bayesian crash rates were  examined for stability of parameter estimates, fit statistics, number and  types of violations that were predictive, predictive power and future  predictive power. Some of the diagnostics cannot be compared in  absolute terms, but rather should be compared across models fit to the  same data. For example, the AIC must be compared across competing  models fit on the same data.

The crash status (yes/no) model was evaluated in the out-of-sample  validation data, where each model was re-fit on the validation sample,  and the diagnostics were examined and compared to those from the  model-building sample. As an additional sensitivity analysis, the same set  of inputs for each of the model groups one through four were also fit using  a Bayesian crash rate outcome, via a linear regression fit to the model- building sample. Results were compared.

section_title: Model Results
paragraphs: Since diagnostics will differ according to the outcome measure, crash  status (yes/no) versus crash rate, information for these outcome types is  displayed separately. For results of models for the crash status (yes/no),  see tables 9 and 10. For results for the Bayesian crash rates, see table  11. Given that a high value of the H-L p-value (close to 1) indicates good  model fit, according to this measure, most of the models fail to fit  acceptably, and none of the models fit well.

Within the same data, a lower value of the AIC indicates better fit;  therefore, the stepwise models perform best, and do nearly as well  regarding the ROC and generalized R-squared when compared to the  more complicated full model. But even for the stepwise models, the ROC  and R-squared do not indicate a strong predictive relationship. This  finding is echoed by the number of effects in the model, relative to the  number of potential violations (about 160) and the number of stable  effects.

One aspect of predictive power is the ability for a model to discriminate  the observed outcomes based on model predictions. Classification tables  describe a model’s classification accuracy with correct and incorrect  classifications, as measured by sensitivity (correctly predict an event) and  specificity (correctly predict a non-event), and false positive (incorrectly  predict a non-event) and negative rates (incorrectly predict an event).

Classification tables for the simple, full, and stepwise model within a  model group are presented in table 10. The observed proportion of  crashes, approximately 0.2 for the unrestricted data and 0.66 for the data  restricted to carriers with more than 20 vehicles, is used as the cut-point  to classify predicted probabilities for a carrier into a predicted event  (crash) versus non-event (no crash). The predicted crash status for a  particular model is compared to the actual post-crash status, resulting in a  series of table rows, one for each model, that examine the false positives,  false negatives, and other quantities that help evaluate the predictive  quality of a model.

For unrestricted data, the false negative rate (or the rate that results from  incorrectly classifying a carrier to a non-alert status), is relatively low  (around 11 percent) compared to the false positive rate (ranges from  about 56 to 58 percent). This is a desired result if it is considered more  appropriate to be conservative and put a carrier in alert status, even if that  alert status is incorrect (false positive), compared to misclassifying a  carrier into non-alert when an alert would be called for (false negative).  The restricted data have a higher false negative rate (from 42 to 44  percent) than false positive rate (around 14 to 19 percent), and this false  negative rate is also higher than the full data false negative rate. For the  restricted data with higher false negative rates, this means a higher  percentage of carriers are being classified in non-alert when they have  crashed than the percent classified as alert, but that did not crash, and  such a scenario is not desirable under a conservative preference toward  low false negative rates. In addition, the sensitivity and specificity are both  moderate at best within data (restricted versus full), further evidence of  the inability for models to discriminate.

To address whether crash status (yes/no) has a different relationship with  violations than the crash rate, we compare conclusions of crash status  (yes/no) versus crash rate models. Examining sensitivity to the prediction  of crash status (yes/no) versus crash rate, the stepwise selected model  will be compared to logistic regression results for the model-building and  the validation sample (see Table 11).model indicates that the numbers of effects that are related to crash rate  are small, and that the better fitting models tend to have only a few  predictors included. Specifically, Mallow’s Cp statistic indicates a model is  preferable when Cp is around or smaller than the number of effects (p),  and the model is more parsimonious than competing models. The model  fit to the restricted data, where carriers have greater than 20 vehicles,  (stepwise model number 22), includes only 34 stable effects, and 72  effects altogether, but the model fit is more stable (i.e., relatively fewer  unstable effects) and has the best (lowest) Cp, while also having similar  explained variance and low AIC. However, it is interesting to note that the  simple model, model 21, performs similarly according to some measures,  such as Root MSE and R-squared, though this model does not contain  violation rate information.

section_title: Model Predictive Power
paragraphs: Comparing how well the models perform when applied to the validation  sample that consists of new observations——which are not included in  the model-building sample—informs the precision of SMS with respect to  predicting crashes. We examine the number of violations and the violation  types that are included across the model groups (logistic and linear) and  sub-models (stepwise and full). We compare this to the number of models  within which each violation was found to be a significant and a stable  predictor of crash outcomes. Importantly, of the reduced set of  approximately 160 violations considered, only 13 violations were  significant in at least half of the 24 models that incorporate violations (i.e.,  stepwise and full models).

There were 10 different possible models for the logistic model-building  sample, and these were also evaluated on the validation sample and on  the model-building sample, but with a linear regression setting, resulting  in 30 possible models. However, we regarded only 24 of these 30 models  as informative since we exclude the 6 simple models that ignore the pre- violation information. Of the violations considered, only speeding  (violation 3922S) and failure to use a seatbelt while operating CMV  (39216) were significant and stable in all 24 models. A similar picture  arises for some other violations, though many of the models did not result  in a significant relationship between the violation in question and the  crash outcome, as indicated in table 12. Only 41 violations were  significant in 5 or more models out of 24. However, even for the top 13  violations with respect to frequency of significance and stability across the  24 models, predictive power is still affected by poor model diagnostics.  This is echoed in the results from the predictive relationship when  compared to the linear regression model for Bayesian crash rates (results  in table 11), where the model that excluded all violations performed  similarly to models that included some significant violations. Whether  modeling crash status (yes/no) or a crash rate, the predictive power of  SMS violations is weak.

When comparing the predictive power of the models that result from the  model-building sample, once applied to the validation sample, there is a  consistent picture regarding the model fit (see table 13). In particular, the  model fit is generally poor according to the H-L value; the stepwise model  tends to perform better according to the AIC, but the ROC, adjusted R2,  and percent discordant do not indicate the models have a strong ability to  discriminate and predict future crashes. Classification tables that result  from evaluating the model-building sample models, but estimated from  the validation sample, generally resulted in similar results to those  presented in table 10.

section_title: Conclusions
paragraphs: The predictive power observed in these modeling and sensitivity analyses  indicates that SMS may be less precise than what is reported and that the  available information on violations is limited for the purpose of scoring  carriers or predicting their crash risk. Regardless of which type of model  we fit, we see that the predictive power of our models is low, and the use  of the SMS violations in predicting future crashes is not very precise. The  number of stable and significant effects across the various model-fitting  scenarios that include violations is small. For the about 800 violations in  SMS, only around 160 met the basic criteria of non-zero variance and  non-zero counts for at least 1 percent of the sample. Of these, only two  violations (speeding and failure to wear a seatbelt while operating a CMV)  consistently appeared as a stable predictor of crashes, regardless of data  and model. While some other violations appeared in models, only 13  were significant and stable in at least half of the models, most were  significant in no more than half the models examined, and most often in  fewer than 5 of the models. The results did not vary substantially  according to whether observed versus Bayesian violation rates, crash  versus Bayesian crash rates, or restricted data (carriers with more than  20 vehicles) versus full data were used to estimate crashes. Therefore  the modeling attempts did not overcome the issues that result from small  exposures. The results were generally confirmed when evaluated on a  validation sample, indicating the future prediction is stable, yet not strong.  Ultimately, much of the variance in crash predictions remains  unexplained, regardless of the model and model-building data, so that the  SMS might be less precise when the objective is to predict crashes.

section_title: Appendix VI: Descriptive Statistics on Motor Carrier Population and Results of GAO’s Analysis
paragraphs: This appendix provides additional information and illustrations of the  distribution of motor carrier population included in our analysis such as  carrier size, number of crashes, inspections, and high risk status (see  table 14). It also provides results of our analysis on the number and  percentage of carriers above or below intervention thresholds, as well as  the frequency and rate of crashes for each of those groups of carriers  within each BASIC using FMCSA’s methodology and the illustrative  alternative methodology (i.e., using a stronger data sufficiency standard)  demonstrated earlier in the report. In addition, this appendix provides  summary statistics of the various motor carrier populations used in  FMCSA and GAO analysis. These statistics include, among other things,  the numbers of carriers with an SMS score (i.e., “measure”) and the  number of carriers above an intervention threshold in at least one BASIC.  Finally, this appendix provides the complete graphical results of our  analysis of FMCSA’s violation rates, safety event groups, and distribution  of SMS scores for carriers above FMCSA’s intervention threshold using  FMCSA’s methodology.

Table 15 contains the results of our analysis using FMCSA’s SMS 3.0  methodology. This analysis calculated the number and percentage of  carriers above and below intervention thresholds for each BASIC using  carrier data from December 2007 through December 2009, and  determined which carriers subsequently crashed during the 18-month  evaluation period, December 2009 through June 2011. The analysis also  presents aggregate crash rates for comparison purposes.

Table 16 contains the results of our analysis using an illustrative  alternative incorporating a stronger data sufficiency standard, among  other things, as described elsewhere in this report (e.g. carriers with 20 or  more inspections or 20 or more vehicles, depending upon the BASIC). As  in the previous table, this analysis calculated the number of carriers  above and below intervention thresholds for each BASIC using carrier  data from December 2007 through December 2009, and determined  which carriers subsequently crashed during the subsequent 18-month  period, December 2009 through June 2011. The analysis also presents  aggregate crash rates for comparison purposes.

Table 17 contains selected SMS outcomes based on results reported by  FMCSA’s and from GAO’s analysis.

The following figures are graphical results of our analysis of the average  and range of violation rates for carriers, percentage of carriers above  FMCSA’s intervention thresholds for various safety event group  categories, and distribution of SMS scores for carriers above FMCSA’s  intervention thresholds using FMCSA’s methodology as discussed in the  body of this report above. Figures 10 through 16 contain the average and  range of violation rates for all carriers (where a violation rate could be  calculated) by carrier size, for all the BASICS. Figures 17 through 25  contain the percentage of carriers above intervention thresholds within  safety event groups for each BASIC. Finally, figures 26 through 32 show  the distribution of carriers above intervention thresholds for each BASIC  by carrier size.

section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, H. Brandon Haller, Assistant  Director, Russell Burnett, Melinda Cordero, Jennifer DuBord, Colin Fallon,  David Hooper, Matthew LaTour, Grant Mallie, Jeff Tessin, Sonya  Vartivarian, and Joshua Ormond made key contributions to this report.


Question-summary annotations: questions: question: How does SMS related to FMCSA?
answer: A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers.
child_questions: question: What is one challenge that FMCSA has faced?
answer: First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers.

question: What is another challenge with SMS?
answer: Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores.
child_questions: question: To what extent has FMCSA addressed this problem?
answer: FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores.
child_questions: question: What was the outcome of this predictability issue?
answer: As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.

questions: question: Why is FMCSA's methodology limited?
answer: FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores.
child_questions: question: How can FMCSA improve its methodology?
answer: GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes.
child_questions: question: What are the tradeoffs of this approach?
answer: This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.

questions: question: What does FMCSA with SMS scores?
answer: In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate.
child_questions: question: What would be necessary in order for FMCSA to use this information to assess carriers?
answer: Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used.

question: What is the mission of FMCSA?
answer: Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities.
child_questions: question: How can FMCSA better achieve this goal?
answer: GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.

questions: question: What is the scale of commercial truck and bus crashes?
answer: From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities.

question: How does FMCSA identify the riskiest vehicles?
answer: In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program.
child_questions: question: What is the purpose of CSA?
answer: CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations.
child_questions: question: How does CSA operate?
answer: CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.

questions: question: What was GAO directed to do?
answer: GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA.

question: What does this report examine?
answer: This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers.

question: How did GAO come to understand SMS?
answer: GAO spoke with FMCSA officials and stakeholders to understand SMS.

question: What methodology did GAO use?
answer: Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores.
child_questions: question: What else did GAO do?
answer: GAO also evaluated SMS's ability to predict crashes.

questions: question: What does GAO recommend?
answer: GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers.
child_questions: question: What should be taken into account when determining operationability?
answer: In addition, determination of a carrier's fitness to operate should account for limitations in available performance information.

question: How did GAO respond to comments from USDOT?
answer: In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations.
child_questions: question: How did USDOT respond to GAO?
answer: USDOT agreed to consider the recommendations.