Federal Motor Carrier Safety: Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers
Released: 2014-02-03T00:00:00

Letter

Background

Motor Carrier Industry Diversity

The commercial motor carrier industry represents a range of businesses, including private and for-hire freight transportation, passenger carriers, and specialized transporters of hazardous materials. As of 2012, FMCSA estimates that there were more than 531,000 active motor carriers, a number that fluctuates over time due to the approximately 75,000 new applications that enter the industry each year combined with thousands of carriers annually leaving the market. Among carriers we assessed for this report, most that operate in the United States are small firms; 93 percent of carriers own or operate 20 or fewer motor vehicles. Nonetheless, a large percentage of vehicles on the road are operated by large carriers. Approximately 270 carriers have more than 1,000 vehicles each and account for about 29 percent of all vehicles that FMCSA oversees.

FMCSA’s Role

FMCSA is responsible for overseeing this large and diverse industry. FMCSA establishes safety standards for interstate motor carriers as well as intrastate hazardous material carriers operating in the United States. To enforce compliance with these standards, FMCSA partners with state agencies to perform roadside inspections of vehicles and investigations of carriers. In fiscal year 2012, FMCSA had a budget of approximately $550 million and more than 1,000 FMCSA staff members located at headquarters, four regional service centers, and 52 division offices.

In 2008, FMCSA launched an operational model test of CSA in four states and began implementing the CSA program nationwide in 2010. CSA is intended to improve safety beyond the prior SafeStat program by identifying safety deficiencies through better use of roadside inspection data, assessing the safety fitness of more motor carriers and drivers, and using less resource-intensive interventions to improve investigative and enforcement actions. From fiscal year 2007 through fiscal year 2013, FMCSA obligated $59 million to its CSA program, including CSA development and technical support, information technology upgrades, and training. For fiscal year 2014, FMCSA requested $7.5 million for CSA.

CSA has three main components:

Safety Measurement System. SMS uses data obtained from federal or state roadside inspections and from crash investigations to identify the highest risk carriers. SMS was designed to improve on SafeStat by incorporating all of the safety-related violations recorded during roadside inspections. Carriers potentially receive an SMS score in seven categories based on this information. Intervention. A set of enforcement tools, such as warning letters, additional investigations, or fines are used to encourage the highest risk carriers to correct safety deficiencies, or place carriers out-of- service.

Safety Fitness Determination Rule. This future rulemaking will amend regulations to allow a determination—based in part on some of the same information used to calculate SMS—as to whether a motor carrier is fit to operate on the nation’s roads.

SMS Carrier Performance

SMS, the measurement system component of CSA, uses the data collected from roadside inspections and crash reports to quantify a carrier’s safety performance relative to other carriers. Specific carrier violations recorded during roadside inspections are assigned to one of six Behavioral Analysis and Safety Improvement Categories (BASIC).

According to FMCSA, these BASICs were developed under the premise that motor carrier crashes can be traced to the behavior of motor carriers and their drivers. A seventh category, called the Crash Indicator, measures a carrier’s crash involvement history (see table 1). Each SMS score is designed to be a quantitative determination of a carrier’s safety performance.

For each of the approximately 800 violations that fall under the various BASICs, FMCSA assigns a severity weight that is meant to reflect the violation’s association with crash occurrence and crash consequence when compared with other violations within the same BASIC. For example, reckless driving violations, categorized in the Unsafe Driving BASIC, are assigned a severity weight of 10 out of a possible 10 because FMCSA determined that these violations have a stronger relationship to safety risk than some other types of violations. Unlawfully parking, by comparison, is also categorized in the Unsafe Driving BASIC, but is assigned a severity weight of 1 out of 10.

FMCSA calculates SMS scores for carriers every month through a process that has three main steps, each of which is made up of several calculations.

Relevant inspections are either a driver inspection, in which the inspection focuses on driver-related requirements, such as the driver’s record of duty or medical certificate, or a vehicle inspection, which focuses on the condition of the motor vehicle. Driver inspections are the relevant inspection for the Unsafe Driving, Hours-of-Service Compliance, Driver Fitness, and Controlled Substances and Alcohol BASICs. Vehicle inspections are considered relevant inspections for the Vehicle Maintenance BASIC. For the Hazardous Materials BASIC, carriers that transport placardable quantities of hazardous materials are also subject to vehicle inspections as the relevant inspections. Throughout the report, we will refer to relevant inspections as simply inspections. another calculation—the number of vehicles a carrier operates adjusted by the number of vehicle miles.

FMCSA accounts for exposure in order to make the scores comparable across carriers. This approach has tradeoffs; while carriers can be compared without penalizing some for having had more inspections or road activity, exposure itself can be considered an element of risk. All else being equal, carriers with more road activity are involved in more crashes and potentially pose more risk to safety.

Step 2: Data sufficiency. Depending on the BASIC, carriers generally receive SMS scores if they meet minimum thresholds of exposure (i.e., number of vehicles or inspections), or a minimum number of inspections with violations (i.e., “critical mass”). For purposes of display on FMCSA’s public website and identifying the highest risk carriers for directing enforcement resources, FMCSA does not include scores for carriers that do not meet a so-called critical mass of violations. For each BASIC, this typically requires a minimum number of inspections that include violations in that BASIC, a violation in that BASIC in the last 12 months, and, for some BASICs, a violation during the most recent inspection.

Step 3: Dividing carriers into peer groups. After calculating violation rates, FMCSA assigns carriers it determines have sufficient exposure to peer groups with similar levels of on-road activity, or what the agency refers to as safety event groups. According to FMCSA, safety event groups are designed to account for the inherent greater variability in violation rates based on limited levels of exposure and the stronger level of confidence in violation rates based on carriers with higher exposure. FMCSA assigns carriers to safety event groups based on their number of inspections, the number of inspections with violations, or crashes the carriers have accrued in the previous 2 years. Within each safety event group, FMCSA calculates SMS scores by ranking carriers’ violation rates (obtained in step 1 above) and assigning each carrier a percentile score ranging from 0 to 100, where 100 indicates the highest violation rate and the highest estimated risk for future crashes. FMCSA displays scores for five of the BASICs on its public website.

Interventions

Once SMS scores are calculated, FMCSA begins a Safety Evaluation that uses SMS scores to identify carriers with safety performance problems requiring intervention. FMCSA has defined a fixed percentage threshold for each BASIC that identifies those carriers that pose the greatest safety risk. (For example, the threshold for the Unsafe Driving BASIC is 65 for most carriers.) These carriers are then subject to one or more FMCSA actions from a suite of intervention tools that were expanded as part of CSA. Tools such as warning letters and on- and off-site investigations allow FMCSA and state investigators to focus on specific safety behaviors. FMCSA can also use enforcement strategies such as fines or placing a carrier out-of-service. The range of available enforcement options gives FMCSA investigators flexibility to apply interventions commensurate with a carrier’s safety performance (see table 2). Seven of the nine interventions are currently implemented nationwide. Prior to CSA, FMCSA investigators’ only tool was a labor intensive, comprehensive on-site investigation. With the additional set of interventions, FMCSA aims to reach more carriers with its existing resources.

According to FMCSA and state safety officials, an investigation or other intervention can also be initiated based on the results of a crash investigation, a complaint against a carrier, or a consistent pattern of unsafe behavior by a carrier. FMCSA further designates some carriers that exceed multiple BASIC thresholds as “high risk.” According to FMCSA, many of these carriers are assigned a Safety Investigator, who must complete a comprehensive review within a year regardless of any changes in the carrier’s score. A carrier is considered high risk if it either: has an SMS score of 85 or higher in the Unsafe Driving BASIC or Hours-of-Service Compliance BASIC or the Crash Indicator, and one other BASIC at or above the intervention threshold, or exceeds the intervention threshold for any four or more BASICs.

Carrier Fitness to Operate

Currently, FMCSA can only declare a carrier as unfit to operate upon a final unsatisfactory rating following an on-site inspection. In addition, FMCSA can order a carrier to cease interstate operations if it determines that the carrier is an imminent hazard. FMCSA can make this determination for several reasons including:

FMCSA determining the carrier to be an imminent hazard. receiving an “unsatisfactory” safety rating during an on-site comprehensive investigation and failing to improve the rating within 45 or 60 days; failing to pay a fine after 90 days; failing to meet the standards required for a New Entrant Audit; or According to FMCSA, during fiscal year 2012, the agency issued 855 out- of-service orders due to an unsatisfactory rating, 1,557 for failing to pay a fine, and 47 because a carrier was determined to be an imminent hazard.

FMCSA has indicated its plans to propose using the same performance data that inform SMS scores to determine whether a carrier is fit to continue to operate. According to FMCSA, the Safety Fitness Determination rulemaking would seek to allow FMCSA to determine if a motor carrier is not fit to operate based on a carrier’s performance in five of the BASICs, an investigation, or a combination of roadside and investigative information. FMCSA proposes doing this through a public rulemaking process; it currently estimates that it will issue a proposed rule in May 2014.

CSA Program Increases Carrier Interventions, but FMCSA Faces Challenges in Identifying High Risk Carriers

CSA has been successful in raising the profile of safety in the motor carrier industry and providing FMCSA with more tools to increase interventions with carriers. However, FMCSA faces two major challenges in reliably assessing safety risk for the majority of carriers in the industry and prioritizing the riskiest carriers for intervention. First, we found that the majority of regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, for most carriers, FMCSA lacks sufficient safety performance information to ensure that FMCSA can reliably compare them with other carriers. FMCSA mitigates this issue by—among other things—establishing data sufficiency standards. However, we found that these standards are set too low, and by strengthening data sufficiency standards SMS would better identify risky carriers and better prioritize intervention resources to more effectively reduce crashes. Setting a data sufficiency standard involves tradeoffs between scoring more carriers and ensuring that the scores calculated are reliable for the purposes for which they are used.

CSA Expands FMCSA’s Reach and Raises the Profile of Safety in the Industry

CSA has helped FMCSA reach more carriers and provided benefits to a range of stakeholders. Since CSA was implemented nationwide in 2010, FMCSA has intervened with more carriers annually than under SafeStat. From fiscal year 2007 to fiscal year 2012, FMCSA increased its number of annual interventions from about 16,000 to about 44,000, largely by sending warning letters to carriers deemed to be above the intervention threshold in one or more BASICs (see table 3). FMCSA and state partners also took advantage of new ways to investigate carriers, such as off-site investigations and on-site focused investigations, to complete 23 percent more investigations in fiscal year 2012 compared to fiscal year 2007 when only compliance reviews were used.

In addition, CSA provides data for law enforcement and industry stakeholders about the safety record of individual carriers. For example, as part of the CSA program, FMCSA publicly provides historical individual carrier data on inspections, violations, crashes, and investigations on its website. According to law enforcement and industry stakeholders we spoke with, CSA organizes violation information for law enforcement and carrier data related to the BASICs help guide the work of state inspectors during inspections.

Law enforcement officials and industry stakeholders generally supported the structure of the CSA program. These stakeholders told us that CSA’s greater reach and provision of data have helped raise the profile of safety issues across the industry. According to industry stakeholders, carriers are now more engaged and more frequently consulting with law enforcement for safety briefings. In Colorado, law enforcement officials told us that CSA has improved awareness and engagement within the motor carrier industry there. A state industry representative told us that CSA has improved safety because carriers are in a competitive business and can feel pressure to improve safety scores to gain an advantage over the competition.

Relationship between Violation of Most Regulations and Crash Risk Is Unclear

The relationship between violation of most regulations FMCSA included in the SMS methodology and crash risk is unclear, potentially limiting the effectiveness of SMS in identifying carriers that are likely to crash. According to FMCSA, SMS was designed to improve on its previous approach to identify unsafe motor carriers by incorporating into the BASICS all of the safety-related violations recorded during roadside inspections. For SMS to be effective in identifying carriers that crash, the violation information that is used to calculate SMS scores should have a relationship with crash risk. Carriers that violate a given regulation more often should have a higher chance of a crash or a higher crash rate than carriers that violate the regulation less often. However, we found that FMCSA’s safety data do not allow for validations of whether many regulatory violations are associated with higher crash risk for individual carriers. Our analysis found that most of the regulations used in SMS were violated too infrequently over a 2-year period to reliably assess whether they were accurate predictors of an individual carrier’s likelihood to crash in the future. We found that 593 of the approximately 750 regulations we examined were violated by less than one percent of carriers. Of the remaining regulations with sufficient violation data, we found 13 regulations for which violations consistently had some association with crash risk in at least half the tests we performed, and only two violations had sufficient data to consistently establish a substantial and statistically reliable relationship with crash risk across all of our tests. (For more information, see app. V.) FMCSA attempted to compensate for the infrequency of violations by, among other things, evaluating aggregate data to establish a broader relationship between a However, evaluations completed by group of violations and crash risk.outside groups have found weaker relationships between SMS scores and the crash risk of individual carriers than FMCSA’s evaluations of aggregate data (for more information, see app. IV). SMS is intended to provide a safety measure for individual carriers, and FMCSA has not demonstrated relationships between groups of violations and the risk that an individual motor carrier will crash. Therefore, this approach of aggregating data does not eliminate the limitations we identified.

Most Carriers Lack Sufficient Information to Reliably Compare Safety Performance across Carriers

Most carriers lack sufficient safety performance information to ensure that FMCSA can reliably compare them with other carriers. As mentioned, SMS is designed to compare violation rates across carriers for the purposes of prioritizing intervention resources. These violation rates are calculated by summing a carrier’s weighted violations relative to each carrier’s exposure to committing violations, which for the majority of the industry is very low. About two-thirds of carriers we evaluated operate fewer than four vehicles and more than 93 percent operate fewer than 20 vehicles. Moreover, many of these carriers’ vehicles are inspected infrequently. (See table 14 in app. VI) Generally, statisticians have shown that estimations of any sort of rate—such as the violation rates that are the basis for SMS scores—become more reliable when they are calculated from more observations. In other words, as observations increase, there is less variation and thus more confidence in the precision of the estimated rate. Given that SMS calculates violation rates for carriers having a very low exposure to violations, such as operating one or two vehicles or subject to a few inspections, many of the SMS scores Carriers with based on these violation rates are likely to be imprecise.few inspections or vehicles will potentially have estimated violation rates that are artificially high or low and thus not sufficiently precise for comparison across carriers. Further, because SMS scores are calculated by ranking carriers in relation to one another, imprecise rate estimates for some carriers can cause other carriers’ SMS scores to be higher or lower than they would be if they were ranked against only carriers with more reliable violation rates. This creates the likelihood that many SMS scores do not represent an accurate or precise safety assessment for a carrier. As a result, there is less confidence that SMS scores are effectively determining which carriers are riskier than others. (App. II provides a more technical discussion of these issues.)

For the five SMS BASICs for which FMCSA uses relevant inspections as a measure of exposure—Hours-of-Service Compliance, Driver Fitness, Controlled Substances and Alcohol, Vehicle Maintenance, and Hazardous Materials—estimated violation rates can change by a large amount for carriers with few inspections even when the number of their violations changes by a small amount. For example, for a carrier with 5 inspections, a single additional violation could increase that carrier’s violation rate 20 times more than it would for a carrier with 100 inspections. This sensitivity can result in artificially high or low estimated violation rates that are potentially imprecise for carriers with few inspections. As an example, our analysis of FMCSA’s method shows that among carriers for which we calculated a violation rate for the Hours-of- Service Compliance BASIC, violation rate estimates are more variable for carriers with fewer inspections. As shown in figure 1, violation rates tend to vary by a larger amount across carriers with few inspections than across carriers with more inspections. As a consequence, a high estimated violation rate for a carrier with few inspections may reflect greater safety risk, an imprecise estimate, or both. Further, comparisons among carriers are meaningful only to the extent they involve carriers with sufficient inspections and thus more precise estimated violation rates.

Similar to carriers with few inspections, carriers with few vehicles are also subject to potentially large changes in their estimated violation rates, which can affect a carrier’s SMS scores. For the Unsafe Driving BASIC and the Crash Indicator, FMCSA measures exposure using a hybrid approach that considers a carrier’s number of vehicles and its vehicle miles traveled—when the latter information is available. Figure 2 shows that among carriers for which we calculated a violation rate using FMCSA’s method for the Unsafe Driving BASIC, carriers that operate fewer vehicles, for example fewer than 5, experience a greater range in violation rates per vehicle than carriers operating more vehicles, for example, greater than 100. (For similar results on other BASICs, see figures 10 to 16 in app. VI.)

Researchers have raised additional concerns about the quality and accuracy of the data FMCSA uses to calculate SMS scores that could potentially compound the problems with the precision of violation rate estimates. These issues further limit the precision of carriers’ estimated violation rates, and consequently their SMS scores. For example:

The frequency of an individual carrier’s inspections varies depending on where the carrier operates. States vary on inspection and enforcement practices. Some studies have shown that inspectors or law enforcement officers in some states cite vehicles for certain violations more frequently than in other states.

Delays in reporting crash data to FMCSA, as well as missing or inaccurate data, can affect a carrier’s Crash Indicator SMS scores. These delays can vary by state.

Data elements used to calculate violation rates for the Unsafe Driving BASIC and Crash Indicator are based on information that is self reported by the carrier. Inaccurate, missing, or misleading reports by a carrier could directly influence their SMS scores. Additionally, among carrier data we evaluated, more than 50 percent did not report their vehicle miles traveled to FMCSA.

FMCSA Has Worked to Address Issues with Precision, but Its Methods Do Not Fully Address Limitations

FMCSA acknowledges that violation rates for carriers with low exposure can be less precise and they attempt to address this limitation in two main ways, but the methods incorporated do not solve the underlying problems. As a result, SMS scores for these carriers are less reliable as relative safety performance indicators, which may limit FMCSA’s ability to more effectively prioritize carriers for intervention.

Data Sufficiency Standards

FMCSA established minimum data sufficiency standards to eliminate carriers that lack what it has determined to be a minimum number of inspections, inspections with violations, or crashes to produce a reliable SMS score. For example, in the Hours-of-Service Compliance BASIC, FMCSA does not calculate SMS scores for a carrier unless it has at least three inspections and at least one violation within the preceding two years. In addition, as previously mentioned FMCSA applies another data sufficiency standard requiring a carrier to have a “critical mass” of inspections with violations in order for an SMS score to be a basis for potential intervention, or to be publicly displayed.

While this approach helps address the problems for carriers with low exposure, it is not sufficient to ensure that SMS scores effectively prioritize the riskiest carriers for intervention. For most BASICs, we found FMCSA’s data sufficiency standards too low to ensure reliable comparisons across carriers. In other words, many carriers’ violations rates are based on an insufficient number of observations to be comparable to other carriers in calculating an accurate safety score. Our analysis shows that rate estimates generally become more precise around 10 to 20 observations, higher than the numbers that FMCSA uses for data sufficiency standards. However, the determination of the exact data sufficiency standard needs to based on a quantitative measure of confidence to fully consider how precise the scores need to be for the (For more information, see purposes for which the scores are used.app. II.)

Safety Event Groups

FMCSA groups the carriers meeting FMCSA’s data sufficiency standards for each BASIC into safety event groups in order to, according to FMCSA, “account for the inherent greater variability in violation rates based on limited levels of exposure and the stronger level of confidence in violation rates based on higher exposure.” based on inspections or inspections with violations depending on the BASIC or on crashes for the Crash Indicator. For example, the first safety event group in the Hours-of-Service Compliance BASIC includes carriers that received from 3 to 10 inspections; the second group includes carriers that received from 11 to 20 inspections, and so forth. Within each safety event group, FMCSA rank orders carriers by violation rate and assigns a percentile as an SMS score.

CSA, CSMS Methodology, Version 3.0.1, Revised August 2013. exceed FMCSA’s intervention thresholds at disproportionately higher rates than carriers with more exposure. For example, FMCSA’s Hours-of- Service Compliance BASIC has five safety event groups. The group of carriers with the fewest number of inspections in each safety event group tends to have a higher percentage of carriers identified as above the intervention threshold than the group of carriers with a greater number of inspections (see fig. 3). This suggests that FMCSA’s methodology is not adequately accounting for differences in exposure, as it is intended to do, but rather is systematically assigning higher scores for carriers with fewer inspections. (See figs. 17 to 25 in app. VI for other BASICs.)

FMCSA’s method of categorizing the carriers into safety event groups for the remaining BASICs also demonstrates how imprecision disproportionately affects small carriers. For the Unsafe Driving and Controlled Substances BASICs, FMCSA forms safety event groups based on the number of inspections with violations. Similarly, for the Crash Indicator, safety event groups are based on a carriers’ number of crashes. By using infractions or crashes to categorize carriers, FMCSA is not addressing its stated intent of having safety event groups account for differences in variability due to exposure. As a result, FMCSA derives SMS scores for the Unsafe Driving BASIC and the Crash Indicator by directly comparing small carriers with greater variability in their violation rates—including many carriers with a violation rate based on one vehicle—to larger carriers for which violations rates can be calculated with greater confidence. We found that among carriers that received an SMS score in Unsafe Driving, carriers with fewer than 20 vehicles are more than 3 times as likely to be identified as above the intervention threshold than carriers with 20 or more vehicles (see fig. 4). Of the carriers operating one vehicle, nearly all were identified as above the intervention threshold. (See figs. 26 to 32 in app. VI for other BASICs.)

FMCSA contends that these results are expected because only small carriers that exceed critical mass standards receive an SMS score, and small carriers that exceed this threshold have demonstrated several occurrences of risky behavior despite their limited exposure. However, this illustrates the volatility of rates and the disproportionate effect a single violation can have given how FMCSA has structured SMS. For example, using FMCSA’s data sufficiency standards, a carrier with one vehicle (forty percent of the carriers in our analysis population have one vehicle) and two inspections with unsafe driving violations does not have sufficient information to be displayed or considered for intervention. However, a single additional violation, regardless of the severity of the violation, would likely mean that the carrier would be scored above threshold and prioritized for intervention. A relatively small difference in the number of violations could change a carrier’s status from “insufficient information”, to “prioritized for intervention” with potentially no interim steps. Conversely, a carrier such as this will have a very difficult time improving its SMS score to be below threshold.

Strengthened Data Sufficiency Standards Can Improve FMCSA’s Ability to Identify High Risk Carriers

Our analysis shows that FMCSA could improve its ability to identify carriers at higher risk of crashing by applying a more stringent data sufficiency standard. As previously discussed, FMCSA uses SMS scores to identify carriers with safety performance problems—those above the threshold in any BASIC—for prioritization for intervention, and considers carriers with SMS scores above the intervention threshold in multiple BASICs as high risk. Overall, SMS is successful at identifying a group of high risk carriers that have a higher group crash rate than the average crash rate of all carriers that we evaluated. However, further analysis shows that a majority of these high risk carriers did not crash at all, meaning that a minority of carriers in this group were responsible for all the crashes. As a result, FMCSA may devote significant intervention resources to carriers that do not pose as great a safety risk as other carriers, to which FMCSA could direct these resources. Given the issues with precision discussed above, we developed and tested an alternative to FMCSA’s method that sets a single data sufficiency standard, based on the relevant measure of exposure—either at least 20 inspections or at least 20 vehicles (depending on the BASIC), and eliminates the use of safety event groups. This approach is designed to illustrate how a stronger data sufficiency standard can affect the identification of higher risk carriers and is not meant to be a prescriptive design to replace current SMS methods.effect that including carriers with low levels of exposure and highly variable violation rates can have on FMCSA’s prioritization of carriers for intervention. Using this illustrative alternative, we found that FMCSA would have more reliably identified a higher percentage of carriers that actually had crashed than when compared to its existing methods. (Apps. I and VI provide more detail on this approach.) Specifically: The result of this analysis demonstrates the

This illustrative alternative identified about 6,000 carriers as high risk.

During the evaluation period of our analysis, these carriers’ group crash rate was approximately the same as the rate for FMCSA’s high risk group (about 8.3 crashes per 100 vehicles). However, a much greater percentage of carriers (67%) identified as high risk using alternative higher data sufficiency standards crashed, and these carriers were associated with nearly twice as many crashes (see table 4).

For five out of six BASICs, the Crash Indicator, and the high-risk designation, the illustrative alternative identified a higher percentage of individual carriers above the intervention threshold that actually crashed compared with FMCSA’s existing method. (See fig. 5.)

Using both FMCSA’s method and the illustrative alternative, for most of the BASICs and the Crash Indicator the carriers identified above the intervention threshold had a higher crash rate (crashes per 100 vehicles) than those below the intervention threshold (see table 5). However, using FMCSA’s method, crash rates for the Controlled Substances and Alcohol BASIC have the opposite, negative association (3.2 crashes per 100 vehicles for carriers above threshold versus 5.2 crashes per 100 vehicles for carriers below threshold), whereas the illustrative alternative produces a positive association (4.7 crashes per 100 vehicles for carriers above threshold versus 3.8 crashes per 100 vehicles for carriers below threshold).

Overall, these results raise concerns about the effectiveness of the existing SMS as a tool to help FMCSA prioritize intervention resources to most effectively reduce crashes. FMCSA’s existing SMS method successfully identified as high risk more than 2,800 carriers whose vehicles were involved in 12,624 crashes. However, FMCSA would have potentially prioritized limited resources to investigate more than 4,000 carriers that did not crash at all. Prioritizing resources to these carriers would limit FMCSA’s ability to reduce the number of overall crashes, resulting in lost opportunities to intervene with the carriers associated with many crashes.

Implementing a stronger data sufficiency standard as presented involves tradeoffs between the number of carriers FMCSA can score, and the reliability of those scores. Our analysis found that by increasing the data sufficiency standards, fewer carriers would receive at least one SMS score (approximately 44,000 carriers in the illustrative alternative versus approximately 89,000 using FMCSA’s method). The carriers assigned an SMS score under the illustrative alternative accounted for 78.2 percent of all crashes during our evaluation period. FMCSA’s existing method scores carriers responsible for about 85.9 percent of all crashes (see table 6). On the other hand, by setting a higher standard for data sufficiency, the illustrative alternative focuses on carriers that have a higher level of road activity, or exposure, to more reliably calculate a rate that tracks violations and crashes over the 2-year observation period. In addition, exposure itself is a large determinant of overall risk, when defined as a combination of threat and consequence, and could be used as a factor to identify carriers that analysis suggest present a higher future crash risk. This is consistent with the results in table 4 above, which show that a larger proportion of the higher risk carriers in the illustrative alternative crashed and were associated with a larger number and proportion of crashes.

Regardless of where the data sufficiency standard is set, using only SMS scores limits risk assessment for carriers that do not have sufficient performance information. Our analysis shows that using FMCSA’s existing method, about 28% of carriers have at least one SMS score, leaving approximately 72% of carriers without any SMS scores—largely due to insufficient information. The illustrative alternative scores fewer carriers—14%, leaving 86% of carriers without any SMS scores. However, according to an FMCSA official, there are other enforcement mechanisms to assess and place unsafe carriers out-of-service, including when a carrier fails to improve from an unsatisfactory safety rating during a comprehensive review, fails to pay a fine, or FMCSA determines a carrier is an imminent hazard. Further, the FMCSA official said carriers that do not receive an SMS score can still be monitored because the officials can initiate investigations and remove carriers based on complaints and other initiatives. For example, FMCSA conducts inspection strike forces targeting unsafe drivers and carriers in a particular safety aspect, such as drug and alcohol safety records. These tools used in conjunction with the performance data, including roadside inspection and crash data, could provide FMCSA with complementary means to assess and target carriers that do not otherwise have sufficient data to reliably calculate SMS scores.

Precision Required in SMS Scores Depends on How They Are Used

The safety scores generated by SMS are used for many purposes, thus the appropriate level of precision required depends on the nature of these applications. According to FMCSA’s methodology, SMS is intended to prioritize intervention resources, identify and monitor carrier safety problems, and support the safety fitness determination process. In setting a data sufficiency standard, FMCSA needs to consider how precise the scores need to be, and a score’s required precision depends on the purposes for which the scores are used.

FMCSA officials told us the primary purpose of SMS is to serve as a general radar screen for prioritizing interventions. However, as discussed above, due to insufficient data, SMS is not as effective as it could be for this purpose. Further, if the same safety performance data used to inform SMS scores are intended to help determine a carrier’s fitness to operate, most of these same limitations will apply. According to FMCSA, the Safety Fitness Determination rulemaking would seek to allow FMCSA to determine if a motor carrier is not fit to operate based on a carrier’s performance in five of the BASICs, an investigation, or a combination of roadside and investigative information. FMCSA has postponed the planned rulemaking until May 2014. However, basing a carrier’s safety fitness determination on limited performance data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion.

In addition to using SMS for internal purposes, FMCSA has also stated that SMS provides stakeholders with valuable safety information, which can “empower motor carriers and other stakeholders…to make safety- based business decisions.” publicly released SMS scores stating that the data are intended for agency and law enforcement purposes, and readers should not draw safety conclusions about a carrier’s safety condition based on the SMS score, but rather the carrier’s official safety rating. Nonetheless, entities outside of FMCSA are also using SMS scores to assess and compare the safety of carriers. For example: FMCSA includes a disclaimer with the

The Department of Defense has written SMS scores into its minimum safety criteria for selecting carriers of hazardous munitions.

FMCSA has released a mobile phone application—SaferBus—that is designed to provide safety information, including SMS scores, for consumers to use in selecting a bus company.

Multiple stakeholders have reported that entities such as insurers, freight shippers and brokers, and others use SMS scores.

Given such uses, it is important that any information about SMS scoresmake clear to users, including FMCSA, the purpose of the scores, their precision, and the context around how they are calculated. Stakeholders have said that there is a lot of confusion in the industry about what the SMS scores mean and that the public, unlike law enforcement, may not understand the relative nature of the system and its limitations.

CSA, CSMS Methodology, Version 3.0.1 Motor Carrier Preview, Revised August 2013.

Conclusions

With the establishment of its CSA program, FMCSA has implemented a data-driven approach to identify and intervene with the highest risk motor carriers. CSA helps FMCSA to reach more carriers through interventions and provides the agency, state safety authorities, and the industry with valuable information regarding carriers’ performance on the road and problems detected during roadside inspections.

GAO continues to believe a data-driven, risk-based approach holds promise and can help FMCSA effectively identify carriers exhibiting compliance or safety issues—such as violations or involvement in crashes. However, assessing risk for a diverse population of motor carriers—many of which are small and inspected infrequently—presents several significant challenges for FMCSA. As a result, the precision and confidence of many SMS scores is limited, a limitation that raises questions about whether SMS is effectively identifying carriers at highest risk for crashing in the future.

As presented in the report, strengthening data sufficiency standards is one of several potential reforms that might improve the precision and confidence of SMS scores. However, strengthening data sufficiency standards involves a trade-off between assigning scores to more carriers and ensuring that those scores are reliable. Our analysis shows how improving the reliability of SMS scores by strengthening data sufficiency standards could better account for limitations in available safety performance information and help FMCSA better focus intervention resources where they can have the greatest impact on reducing crashes. In addition, if these same safety performance data are going to be used to determine whether a carrier is fit to operate, FMCSA needs to consider and address all identified data limitations, or these determinations will also be at risk.

Recommendations for Executive Action

To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to take the following two actions: Revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers; in doing so, conduct a formal analysis that specifically identifies: limitations in the data used to calculate SMS scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments, and limitations in the resulting SMS scores including their precision, confidence, and reliability for the purposes for which they are used.

Ensure that any determination of a carrier’s fitness to operate properly accounts for limitations we have identified regarding safety performance information.

Agency Comments

We provided a draft of this report to the USDOT for review and comment. USDOT agreed to consider our recommendations, but expressed what it described as significant and substantive disagreements with some aspects of our analysis and conclusions. USDOT’s concerns were discussed during a meeting on January 8, 2014, with senior USDOT officials, including the FMCSA Administrator. Following this meeting, we made several clarifications in our report. In particular, FMCSA understood our draft recommendation to be calling for specific changes to its SMS methodology. It was not our intent to be prescriptive, so we revised our first recommendation to state that FMCSA should conduct a formal analysis to inform potential changes to the SMS methodology. In addition, we clarified in the analysis and conclusions our meaning of reliability in context of the purpose for which SMS is used.

We are sending copies of this report to relevant congressional committees and the Secretary of Transportation. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII.

Appendix I: Scope and Methodology

This report addresses the effectiveness of the Compliance, Safety, Accountability (CSA) program in assessing safety risk for motor carriers. To assess how effectively CSA assesses the safety risk of motor carriers, we reconstructed the models the Federal Motor Carrier Safety Administration (FMCSA) uses to compute the SMS scores for all six Behavior Analysis and Safety Improvement Categories (BASICs) and the crash indicator. We then assessed the effect of changes to key assumptions made by the models. Using data collected by the U.S. Department of Transportation’s Motor Carrier Management Information System (MCMIS) and historical SMS scores, and referencing the SMS algorithm and methodological documentation, we replicated the algorithm for calculating the SMS BASIC scores for the SMS 3.0 methodology. Reconstructing FMCSA’s models and replicating the SMS scores FMCSA produced for carriers was a necessary step to ensure that we understood the complexities of the models, the data used in the calculation of the SMS scores, and that the results we present in this report are comparable to FMCSA’s outcomes. To corroborate our models with FMCSA’s, we compared the SMS violation rates (measure scores) to FMCSA’s results for December 2012. We assessed the reliability of data used, for our purposes, by reviewing documentation on FMCSA’s data collection efforts and quality assurance processes, talking with FMCSA and Volpe National Transportation Systems Center officials about these data, and checking the data for completeness and reasonableness. We determined that the data were sufficiently reliable for the purpose of our data analysis.

We established a population of about 315,000 carriers for analysis that were under FMCSA’s jurisdiction and showed indicators of activity over a 3 and a half year analysis period from December 2007 through June 2011. The criteria used to identify these carriers were:

U.S.-based carriers; interstate or intrastate hazardous materials carriers; carriers with at least one inspection or crash during the 2-year analysis observation period (December 18, 2007 to December 17, 2009); and carriers with a positive average number of vehicle count at any point during the analysis observation period (December 18, 2007, to December 17, 2009) and at any point during the evaluation period (December 17, 2009, to June 17, 2011).

During the first 2 years of this period, December 2007 through December 2009, we used each carrier’s inspection, crash, and violation history to calculate SMS scores. This period is referred to as the observation period. The remaining 18 months, December 2009 through June 2011, were classified as the evaluation period. We used data from this period to identify carriers involved in a crash and estimate crash rates for these carriers. For the approximately 315,000 carriers in our analysis, there were approximately 120,000 crashes during the evaluation period. We chose the lengths of time for observation and evaluation, in part, to match FMCSA’s effectiveness testing methods.

We tested the effectiveness of SMS by identifying and making changes to key assumptions of the model. Given FMCSA’s use of these scores as quantitative determinations of a carrier’s safety performance, we assessed the reliability of SMS scores as defined by the precision, accuracy, and confidence of these scores when calculated for carriers with varying levels of carrier exposure—measured by FMCSA as either inspections or an adjusted number of vehicles. We tested changes to the following characteristics of the model: the SMS measures of exposure, the method used to calculate time weights, the organization of the violations to the six BASICs, and the data sufficiency standards. To evaluate the results produced by each model, including FMCSA’s, we examined the SMS scores and classifications of carriers into the high risk group. We compared the results from our revised models to the results from a baseline model, SMS 3.0. For each model, we measured whether carriers were involved in a crash, calculated group crash rates, and calculated total crashes in the evaluation period for carriers that were and were not classified as high risk in the observation period. Due to ongoing litigation related to CSA and the publication of SMS scores, we did not assess the potential effects or tradeoffs resulting from any public use of these scores.

To determine the extent to which CSA identifies and intervenes with the highest risk carriers, we examined how our changes to FMCSA’s key assumptions affected the safety scores and identification of high risk carriers. Specifically, we identified the carriers with SMS scores above FMCSA’s intervention threshold in each BASIC and the carriers considered high risk according to FMCSA’s high risk criteria. Using this analysis, we designed an illustrative alternative method that incorporates the following changes: including only carriers with at least 20 observations in the following measures of exposure: driver inspections when calculating scores for the Hours-of- Service Compliance, Driver Fitness, and Controlled Substances BASICs; vehicle related inspections for the Vehicle Maintenance BASIC; vehicle related inspections where placardable quantities of hazardous materials are being transported for Hazardous Materials BASIC; and average power units for the Unsafe Driving and Crash Indicator assigning an SMS score to any carrier meeting these data sufficiency standards (e.g., 20 inspections), even if that carrier does not have any violations, was free of violations for 12 months, or had a clean last inspection; eliminating safety event groups because of the stricter data sufficiency using only the average number of vehicles as the measure of exposure for carrier’s assessed in the Unsafe Driving and Crash Indicator BASICs.

Appendix VI provides the complete results of our replication of FMCSA’s existing SMS and our illustrative revision to it.

We also examined the extent to which the regulatory violations that largely determine SMS scores can predict future crashes. We developed eight model groups to test the relationship between violations and violation rates, and crashes. We tested only the violations that had non- zero variance and observations for at least 1 percent of the test population. To control for small exposure measures when estimating rates, we estimated models comparing carriers’ observed crash status to Bayesian crash rates; used observed violation rates versus Bayesian violation rates; and compared a full model sample to a restricted model sample of carriers with at least 20 vehicles.sensitivity analysis to validate the predictive power of the models we developed. We ran multiple variations of these models to determine the number and types of violations that were predictive versus unstable. For We also conducted a more information on this specific analysis and model results, please see appendix V.

In addition, we spoke with FMCSA officials in Washington, D.C., and at the Western Service Center and the Colorado Division Office in Lakewood, Colorado, and reviewed existing studies and stakeholder concerns about the SMS model and its outcomes. To understand the impact of CSA on law enforcement, we spoke with law enforcement officials at the Colorado State Patrol. We selected Colorado because it was one of the initial pilot states for CSA, and has been implementing the program since early 2008. We also interviewed representatives from industry and safety interest groups from the Colorado Motor Carriers Association, the Commercial Vehicle Safety Alliance, and the American Trucking Associations. Additionally, we attended meetings of the Motor Carrier Safety Advisory Committee’s CSA subcommittee and reviewed the minutes and related documentation from other meetings we did not attend. We also reviewed congressional testimony from industry and safety interest representatives from a September 2012 hearing for the House Transportation and Infrastructure Committee. We reviewed stakeholder comments submitted between March 2012 and July 2012 in response to FMCSA’s planned improvements to SMS.

We conducted this performance audit from August 2012 to February 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Estimating Rates of Regulatory Violations in the Safety Measurement System

The FMCSA Safety Measurement System (SMS) methodology involves the calculation of weighted violation rates for regulations within each of six Behavioral Analysis and Safety Improvement Categories (BASICs) and a given time period. (A seventh indicator measures weighted crash rates in previous time periods, or “crash history.”) Carriers are assigned to Safety Event Groups based on measures of their exposure to committing violations, such as the number of driver or vehicle inspections, depending on the BASIC, and the weighted violation rates are transformed into percentiles for carriers within the same group. These percentiles ultimately determine carriers’ alert or high-risk statuses. Because regulatory violation rates strongly influence SMS scores, the precision with which these rates can be calculated becomes important for developing reliable measures of safety, as we discuss in the body of this report.

In this appendix, we summarize statistical methods for estimating rates and assessing their precision, or sampling error. We use these methods to estimate crash rates and their sampling error for a population of motor carriers that were active from December 2007 through December 2009. Carriers may vary widely in their level of activity, known as “exposure.” Both statistical theory and our analysis show that the precision of estimated rates for carriers with low exposure, measured by vehicles or inspections, is lower than for carriers with more exposure, and that rate estimates can become distorted to artificially low or high values for these low-exposure carriers. These results support our findings in the body of this report on the precision of FMCSA’s current approach to calculating safety risk scores and setting data sufficiency standards.

Statistical Methods for Estimating Violation Rates and Their Sampling Variance

Estimating rates of regulatory violations requires data on the number of violations that carriers incur within a given time period. If one makes the assumption that the number of violations is proportional to some measure of exposure (activity) and also assumes that the probability of observing violations within a large number of small independent exposure periods is small, the sampling error of a rate estimate decreases as exposure increases.

Specifically, assume that each carrier in a population of interest has a unique violation rate, λ. For a fixed time period and known exposure, t, the number of violations, V, is distributed as V ~ Poisson (λ t), with E(V) = Var(V) = λ t. Since λ is unknown, it must be estimated from data on regulatory violations and exposure.

The maximum likelihood (ML) estimator for a single carrier’s λ, given the model above, is λı� = v / t, with Var(λı�) = λı� / t = v / t.the rate estimate increases exponentially as exposure decreases. Accordingly, an estimated rate for a specific carrier and time period can vary substantially from λ, particularly when exposure is low.

SMS is primarily concerned with measuring how regulatory violation rates vary over a population of active motor carriers. Even though ordinary methods of estimating these rates are unbiased and consistent, the collection of estimated rates for the population, 𝛌� = {λestimates, such as the percentiles that SMS uses to place carriers into alert and high-risk status, may be similarly prone to error.

Empirical Bayesian methods correct for this problem by estimating λı� for each carrier to better estimate the distribution of rates across a population. Bayesian methods prevent estimates from converging to artificially extreme values for carriers whose raw rate estimates are based on small samples (low exposure). The estimator does this by effectively “borrowing information” from other, larger carriers whose rates can be estimated more precisely. In the evaluation of the CSA Pilot Test for FMCSA, the University of Michigan Transportation Research Institute used empirical Bayesian rate estimation methods to evaluate the association between SMS scores and crash risk, and cited similar benefits to those we discuss here.

For example, see Roger J. Marshall, “Mapping Disease and Mortality Rates Using Empirical Bayes Estimators,” Journal of the Royal Statistical Society, Series C (Applied Statistics) 40, no. 2 (1991): 284, or J. N. K. Rao, Small Area Estimation (Hoboken, NJ, 2003), 206.

Specifically, assume that regulatory violation rates over a population of carriers are distributed as λı� ~ Gamma(α, β), the prior distribution of the parameter of interest. Parameter values for the prior distribution can be assumed, based on historical data on the population of interest, or estimated using a particular sample. Conditional on these rates, the data on regulatory violations are distributed as V | λ , t ~ Poisson(λ t), and the posterior distribution for a specific carrier is given by λ | v, t ~ Gamma(α + v, β + t) (1)

Since the mean of a Gamma variate is α / β and the variance is α / β, the posterior mean and variance of the rate for a given carrier are given by E(λ | v , t) = (α + v) / (β + t) (2)

Var(λ | v , t) = (α + v) / (β + t), and the mean of the prior distribution, α / β. When enough data are available, as indicated by a large exposure term relative to the violation term, the estimate converges to the ordinary, carrier-specific rate estimate. When exposure is low, however, the method combines data from the specific carrier with the mean rate for all carriers.

The variance of Bayesian rate estimates decreases with increased exposure, similar to the variance of ordinary rate estimates. Figure 6 shows how hypothetical rate estimates and 90% posterior intervals for a carrier that experienced 5 crashes vary with the carrier’s exposure, as measured by the number of vehicles. (Although we illustrate rate estimation issues using crash rates, we likely would have obtained similar results if we had estimated regulatory violation rates.) As expected, the precision of the estimates decreases exponentially as the number of vehicles increases. The variance is high in the range of 1 to 5 vehicles and begins to decrease less quickly at approximately 20 vehicles, consistent with our discussion in the body of this report and prior evaluations of SMS.

Thresholds in this approximate range are consistent with criteria used by the Centers for Disease Control and Prevention (CDC) to suppress or caveat rate estimates for the purpose of public display. For example, in its compendium of health statistics in the United States, CDC cautions that “hen the number of events is small and the probability of such an event is small, considerable caution must be observed in interpreting the conditions described by the figures.”

Even though the Bayesian estimates do not converge to extremely low or high values when exposure is low, the uncertainty around the estimates remains high. As figure 6 shows, statistical methods for modeling and estimating rates can quantify this uncertainty explicitly, in order to reflect the varying precision of estimates for motor carriers with more or less observed data. Although the amount of uncertainty that is acceptable in practice depends on the purpose of the estimates, both statistical theory and government agencies estimating rates similar to those involved in the calculation of SMS scores have recognized the need to express the uncertainty of these estimates, particularly when the derived from small samples. This contrasts with FMCSA’s approach, which reports SMS scores as safety risk estimates with no quantitative measures of precision.

Applying Rate Estimation Methods to Motor Carrier Data

To illustrate the rate estimation issues discussed above in the context of motor carrier safety, we estimated individual crash rates for a population of motor carriers that were actively operating in each of two time periods, December 2007 through December 2009, and December 2009 through June 2011, as measured in FMCSA’s Motor Carrier Management Information System (MCMIS). An “active” carrier was one that, in each time period, had at least one inspection or crash and had been recorded as a US-based interstate or intrastate Hazmat carrier. This definition resembled the one we used in replicating SMS, as described in the body of this report and appendix I. We obtained these data from the December 2010 and December 2012 MCMIS “snapshot” data files, as well as a historical file of carrier-specific information that covered all snapshots.

We estimated the raw and empirical Bayesian crash rates for each carrier in the first time period, using data on the number of crashes and vehicles for these carriers and the formulas above. We used the “empirical Bayes” version of the rate estimator, in which the parameters of the prior distribution were estimated from the data. Specifically, we fit the observed rate data for all carriers in the first time period to the negative binomial distribution, parameterized with exposure measured by number of vehicles, and estimated α and β using standard methods of maximum likelihood estimation. The final rate estimates for each carrier were a combination of these parameter estimates and carrier-specific data, according to equation 2 above.

As theory would predict, Bayesian methods prevented crash rates from converging to zero or extremely high values for carriers with low exposure. The left half of figure 7 presents the raw crash rates for our analysis carriers, while the right half presents the empirical Bayesian estimates. The raw estimates for carriers with about 1 to 10 vehicles can be 10 to 20 times higher than for carriers with more than 10 vehicles. In addition, the raw rates cluster at zero for a large number of carriers, particularly for those with low exposure. An underlying crash rate of zero is implausible for active carriers. In contrast, the Bayesian rate estimates are more stable, with no inflation or deflation to extreme values. Since the body of this report finds that 93 percent of carriers in our replication of SMS had fewer than 20 vehicles, Bayesian methods may provide more stable estimates for many specific carriers and may better approximate the distribution of rates across carriers.

In addition to stabilizing rates for small carriers, Bayesian rate estimation methods provide an explicit measure of precision for each carrier’s rate, regardless of size. In figure 8, we show the Bayesian rate estimates for a random sample of 109 carriers in the first period of our analysis population, along with 90 percent Bayesian posterior intervals. (We present these results for a sample to make the intervals readable.) The posterior interval expresses the range over which the true rate exists with a 90 percent probability. Consistent with theory, the precision of the rate estimates increases with exposure—in this case, the number of vehicles. These results apply to actual carriers in the sample, but the results are consistent with those expected by theory. The width of the posterior intervals does not decrease monotonically, however, because the relative number of crashes also affects the variance and is not held constant in the plot.

Appendix III: Evaluating the Statistical Validity of the Safety Measurement System

In this appendix, we express the Safety Management System (SMS) as a statistical measurement model, in order to make its assumptions explicit, and describe how estimating the model could validate those assumptions. We find that FCMSA’s SMS makes a number of strong assumptions about motor carrier safety that empirical data cannot easily validate.

The SMS uses administrative data on inspections of commercial motor carriers, violations of regulations, and crashes to measure carrier safety. Statisticians and other researchers have developed methods to validate measures of such broad concepts as safety, referred to as “latent variables,” using empirical data. These methods are known as “measurement models.” For example, mental health professionals have created scales to measure the existence of broad disorders, such as depression, by combining responses to multiple items on patient questionnaires. SMS has a similar goal: to create scales to measure motor carrier safety risk on several dimensions, such as “Unsafe Driving” or “Vehicle Maintenance,” by combining violation rate data across multiple regulations. Latent variable measurement methods can assess whether these broader measures are valid and reliable, and whether the empirical indicators that go into them actually measure the intended concepts. Estimating the degree to which various indicators measure a broader concept helps confirm and often improve the reliability and validity of the scales constructed.

Structure and Assumptions of SMS

Much of the SMS involves calculating weighted regulatory violation rates for motor carriers in a given time period. FMCSA assigns weights that, in principle, reflect the violations’ associations with one of six dimensions of safety, known as Behavioral Analysis and Safety Improvement Categories (BASICs), such as “Unsafe Driving” and “Vehicle Maintenance.”

The weights represent what FMCSA considers to be the strength of each violation’s association with safety, relative to other violations in the same BASIC. All violations that are categorized in a BASIC get a positive weight ranging from 1 to 10, which implies that they have some association with safety. These weighted violation rates strongly influence the final SMS measures of safety on these dimensions. Each BASIC is linked to a set of violations, which are all assumed to measure the same dimension of safety. Each violation maps to exactly one BASIC, though BASICs map to multiple violations in their associated groups. .

Vij measures the number of times that carrier i violated regulation j in a given time period. 𝜆𝑗 is a weight for each violation. It is the product of a “severity” weight, measuring what FMCSA considers the violation’s “crash risk relative to the other violations comprising the BASIC measurement,” in addition to outcomes thought to be particularly severe (e.g., out-of- service violations), and a time weight, measuring what FMCSA considers the importance of violations from different time periods to estimating a carrier’s current level of safety. By defining Vij for fixed time periods, such as 6 or 12 months prior to the measurement time, we collapse the separate weights used in SMS into 𝜆𝑗, in order to simplify the notation.

Lastly, T measures exposure to committing violations in the time period, which is either a function of carrier’s vehicles and vehicle miles traveled (VMT) or the time-weighted sum of relevant inspections, depending on the BASIC.

SMS transforms the weighted violation rates for each carrier into percentile ranks, after applying a number of “data sufficiency standards” to exclude carriers with few violations, inspections, and/or vehicles. Carriers with percentiles that exceed established thresholds are “alerted” on the relevant BASICs and, if enough alerts or other conditions exist, are identified as “high risk.” As a result, the ultimate measures of safety risk are ordered groups, with cut-points defined by BASIC percentiles for carriers that meet FMCSA’s standards for data sufficiency.

SMS as a Latent Variable Measurement Model

The SMS can be viewed as an attempt to measure latent concepts of “safety,” such as “Unsafe Driving” or “Vehicle Maintenance,” using observed data on regulatory violations and the opportunity to commit them (exposure). Consider the latent variable measurement model below, using notation from a prominent textbook: The weights describing the relationship between the latent and observed The model assumes that a vector of 𝑘=∑ 𝑛𝑔𝑝𝑔 observed variables, r, are determined by p latent variables, 𝜉, and random measurement error, 𝛿. variables make up the block diagonal matrix Λ, with p blocks of weights applied to the corresponding blocks of observed variables. This structure latent variable. In many applications, the model assumes that Cov(𝜉 implies that each group of observed variables is related to exactly one , 𝛿)=0 and E(𝛿 ) = 0 but allows other variances and covariances to be estimated from the data as parameters or fixed to known values.

The SMS is a particular form of the model above. Specifically, SMS defines r as violation rates for k = 826 regulations, where r may include variables measured at different times. It sets p = 6 and relates the violation rates to the BASICs, or latent variables 𝜉 measuring safety, through the weighting matrix Λ. FMCSA created fixed time and severity assumes that 𝛿=0. A graphical version of SMS as a measurement weights for each regulation through a combination of statistical analysis and the opinions of stakeholders. Since SMS is not a stochastic model, it model appears in figure 9 below.

When expressed as a measurement model, the strong assumptions of SMS —and their potential detrimental effect on its usefulness—become clear. FMCSA’s assumption of zero measurement error is unusual for statistical approaches to measurement, given that any particular violation is likely to represent variation in latent variables (in this case, safety) as well as unmeasured variables summarized by the error term. SMS makes specific assumptions about the number of safety dimensions—the latent variables assumed by the model above—as well as their relationships to violation rates. Exactly six dimensions of safety exist (involving regulations), and each violation rate measures only one of them. In other efforts to measure broad concepts using numerous indicators, inference about the existence and relationships among observed and latent variables are endogenous parameters (determined by the model) to be estimated, rather than exogenous parameters (determined outside the model) that are fixed ex ante, ahead of time, as they are here. Finally, SMS takes the unusual step of fixing the values of the weights relating the latent variables measuring safety to violation rates at values other than 0. This assumes a high degree of prior knowledge about the relationships between latent and observed variables. Although FMCSA has conducted several studies of how regulatory violation rates are associated with crash risk, these studies do not directly estimate the degree to which each type of violation reflects one of several dimensions of safety.

One approach to validating the assumptions of SMS is to estimate the parameters of the measurement model above using empirical data on regulatory violation rates. This approach is known as Confirmatory Factor Analysis, which is a special type of measurement model. Because SMS makes specific assumptions about the number of BASICs and the violations that go into them, we can express the system as a measurement model, as discussed above, and estimate the degree to which its assumptions are consistent with reality. For example, SMS assumes that six dimensions of safety exist—labeled BASICs in SMS— and that each violation reflects only one dimension. However, a model that assumes three BASICs and allows violations to reflect multiple dimensions of safety might be a plausible alternative. High violation rates for brake maintenance regulations may indicate worse performance on both the Vehicle Maintenance and Unsafe Driving dimensions of safety. Measurement modeling can identify which of these approaches better fits empirical patterns of regulatory violations. More generally, analyzing SMS as a measurement model can validate its assumptions, such as the values of the severity and time weights, and suggest improvements to better measure safety.

We can extend the SMS measurement model to predict empirical data on crash risk, in order to further validate its ability to identify high-risk carriers. This structural equation modeling (SEM) approach combines the measurement model above with a model that describes how the latent dimensions of safety predict crash risk, generically known as “endogenous observed variables.”

To incorporate outcomes, we extend the measurement model above to assume that the six BASICs are directly related to an empirical measure of crash risk: C measures crash risk; 𝛾 are parameters describing how the latent safety dimensions are related to crash risk; 𝜉 are the safety dimensions; and 𝜀𝑖 parameters describing how the SMS scores relate to crash risk, 𝛾. Strong is a random error term. Estimating this larger model would yield the original parameters of the measurement model, in addition to the correlations between SMS scores and crash risk would further support their ability to identify higher-risk carriers. This is known as “criterion validity” in statistics and social research.

A key strength of this validation approach is that it accounts for the error in measuring broad dimensions of safety when predicting crash risk. Because empirical data on violation rates and SMS scores are indicators of latent concepts of safety, measurement error can distort the underlying relationships between these broader concepts and crash risk. For example, poor vehicle maintenance may be positively associated with higher crash risk, but empirical data on violations of vehicle maintenance regulations may measure both the concept of interest and the enforcement efforts of state and local governments. As a result, the violation rates may be uncorrelated with crash risk simply due to error in measuring the concept of interest. SEM models estimate the relationships among latent variables more precisely by accounting for this measurement error. This contrasts with simpler regression models of crash risk as a function of observed violation rates, which assume that violation rates measure the dimensions of safety without error.

Appendix IV: Prior Evaluations of SMS Scores as Measures of Safety for Specific Carriers and Risk Groups

Previous evaluations of SMS have focused on estimating the correlations between crash risk and regulatory violation rates and Safety Measurement System (SMS) scores. These evaluations have found mixed evidence that SMS scores predict crash risk with a high degree of precision for specific carriers or groups of carriers. This appendix synthesizes the results of these prior evaluations.

Several prior evaluations of SMS have analyzed grouped data, rather than directly analyzing how a carrier’s individual regulatory violation rates and SMS scores predict its own future crash risk. For example, in a pilot evaluation conducted for FMCSA, the University of Michigan Transportation Research Institute (UMTRI) estimated group crash rates within percentiles of SMS scores for each Behavioral Analysis and Safety Improvement Category (BASIC), pooling several hundred carriers in each percentile, to trace out the aggregate relationship between SMS scores and crash risk. Similarly, FMCSA’s Violation Severity Assessment Study analyzed grouped violation data from roadside inspections conducted from 2003 through 2006, in order to compare rates cited in post-crash reports to rates in the general population of carriers.

Ibid., 4-2, 4-6. that did and did not exceed the SMS thresholds to be placed in “alert” or “high risk” statuses.

Aggregate approaches, such as those used in several prior evaluations, do not directly assess the ability of SMS and regulatory violations to predict future crash risk for specific carriers. Well-known findings in statistics on “ecological fallacies” show that associations at higher levels of analysis are not guaranteed to exist at lower levels of analysis. In this application, carriers that crash may have higher violation rates or SMS scores as a group than carriers that do not crash, but this pattern does not necessarily apply to specific carriers within the groups. Because less variation exists at the carrier level, aggregation can overstate the strength and precision of these correlations for individual carriers.

Even when similar correlations exist at the carrier level, comparing average crash rates for SMS percentiles or risk groups does not assess the prediction error for any particular carrier. The average crash rate may be higher for groups of carriers with increasingly high SMS percentiles, but crash rates may vary significantly around these means. This residual variation, not differences in means or other aggregate statistics, is more directly relevant for assessing the quality of predicted crash rates for a particular carrier. In statistical terms, the prediction error summarized by the residual variance of a linear regression model or the classification matrix of a categorical model is what matters for assessing predictive power for individual carriers, not the models’ coefficients, which estimate mean crash rates conditional on these percentiles.

Thus, it is not surprising that previous evaluations of carrier-level data have found weaker relationships between crash risk and SMS scores and regulatory violations than have the evaluations of aggregated data.

UMTRI estimated the relationship between exceeding thresholds in the six non-crash BASICs and mean crash rates, using an empirical Bayesian negative binomial model estimated on carrier-level data. The results showed that carriers exceeding the thresholds for the Unsafe Driving and Vehicle Maintenance BASICs had average crash rates that were 1.1 to 1.8 times higher than carriers not exceeding the thresholds—usually lower than the rate ratios of 1.0 to 5.4 reported by UMTRI’s aggregate analysis and FMCSA’s December 2012 Effectiveness Testing. However, this relationship was negative for the Driver Fitness and Loading/Cargo (currently Hazardous Materials) BASICs, with mean crash rates for alerted carriers that were 0.85 and 0.91 times the rates of non-alerted carriers, respectively. The ratios were not significantly greater than 1 for the Fatigued Driving and Substance Abuse/Alcohol BASICs. Similarly, the American Transportation Research Institute (ATRI) found that alerted carriers in the Unsafe Driving, Vehicle Maintenance, Hours-of-Service, and Controlled Substances/Alcohol BASICs had mean crash rates that were 1.3 to 1.7 times larger than scored carriers not in alert status, but carriers exceeding the Driver Fitness thresholds had mean crash rates that were 0.87 times those of non-alert scored carriers.

Although UMTRI and ATRI analyzed carrier-level data, they validated SMS measures using regression coefficients and similar statistics that describe aggregate correlations. As we discuss above, this approach does not directly quantify predictive power for specific carriers.

Two studies that have directly estimated prediction error for specific carriers, conducted by Wells Fargo Securities and James Gimpel of the University of Maryland, found weaker evidence of the model’s predictive effectiveness. Gimpel found that mean crash rates increased by small amounts as SMS scores increased on the Unsafe Driving, Hours-of- Service, and Vehicle Maintenance BASICs increased.found a similarly positive association for the Unsafe Driving BASIC, but a Wells Fargo negative association for the Hours-of-Service BASIC, in its analysis of 4,600 carriers with at least 25 vehicles and 50 inspections. More critically, the authors showed that scores on these BASICs predict crash rates with a large amount of error, with most R-squared fit statistics ranging from nearly zero to 0.07 for reasonably large analysis samples. Although these studies do not report critical estimates of the residual variance, the R-squared statistics likely imply confidence intervals around predicted crash rates for individual carriers with widths that are several times larger than the predictions themselves. This implies that SMS scores predict future crash risk for specific carriers with substantial error, even though mean crash rates can be higher among carriers with higher SMS scores.

FMCSA used aggregate data to dispute the findings of the Wells Fargo evaluation. Specifically, the agency cited the UMTRI findings that aggregate crash rates were 3.0 to 3.6 times higher for carriers exceeding thresholds for the Unsafe Driving and Hours-of-Service BASICs than for carriers that did not exceed thresholds for any BASIC. In addition, FMCSA highlighted analyses by UMTRI and the Volpe Center of aggregate crash rates across percentiles of SMS scores in the Unsafe and Fatigued Driving BASICs, respectively, which they claimed to show a stronger correlation to crash risk. FMCSA’s approach to evaluating the predictive power of SMS scores resembles its Effectiveness Testing, which compares aggregate crash rates for carriers above and below thresholds for various BASICs.

However, as we discuss above and Wells Fargo discussed in its response to FMCSA, the fact that SMS scores predict aggregate crash rates more strongly at the alert-group or percentile level does not necessarily imply that the scores will predict the crash risk of individual carriers. Recognizing this, the UMTRI evaluation analyzes the data at both the aggregate and carrier levels, and finds that mean crash rate ratios are far smaller at the carrier level than at the alert-group or percentile levels. It should be intuitive that aggregate evidence of effectiveness, stressed in some FMCSA evaluations, shows stronger predictive power than the carrier-level analyses of ATRI, Gimpel, UMTRI, and Wells Fargo. Aggregating violation and crash rates within larger groups effectively increases the sample size used to calculate rates, which reduces their sampling error when compared to the equivalent carrier-level measures. The reduction of sampling error can strengthen the correlations between violation rates and SMS scores and crash risk.

Evaluations of SMS that focus on carrier-level prediction error provide the most appropriate evidence of effectiveness for assessing the safety of individual carriers. FMCSA has stated that one purpose for SMS scores is to predict the future crash risk of individual motor carriers, in order to prioritize resources for intervention and enforcement. In addition, FMCSA reports SMS scores as measures of safety on a public website and the SaferBus Mobile app. To assess the validity of SMS scores for this purpose, evaluations should focus on the system’s ability to predict the crash risk at the carrier level, not its ability to identify groups of carriers with larger crash rates on average or collectively. Measures of predictive accuracy—such as the residual error made when predicting crash rates or the classification error made when assigning carriers to risk groups— are the critical metrics of success, not aggregated crash rate ratios and regression coefficients. When evaluated on these criteria, prior studies show that SMS predicts future crash risk for individual carriers with substantial imprecision.

None of the prior studies has explicitly incorporated measurement error into evaluations of SMS. Since SMS is ultimately a method of creating measures of latent variables, as we discuss in appendix III, the regulations used to calculate scores and the scores themselves have some degree of measurement error. Because existing studies have used statistical methods that assume zero measurement error, more comprehensive attempts to model the measurement structure of SMS and validate its assumptions and predictive power, such as those we discuss in appendix III, may produce different results. The correlations among SMS scores, violation rates, and crash risk may reflect measurement error as much as the underlying relationships among the variables of interest. This more complex analysis is critical for future evaluations of SMS and its ability to measure safety risk.

Appendix V: Analysis of Regulatory Violations and Crash Risk

As a more basic approach to validating SMS, which focuses on the ability of data on regulatory violations in one time period to predict crash risk in a subsequent period, we analyzed the relationship between violation rates and crash risk using a series of statistical models. These models predicted the probability of a crash and crash rates as a function of regulatory violation rates for a population of motor carriers that were actively operating over a recent 3.5-year time period (described below).

We find that a substantial portion of regulatory violations in SMS cannot be empirically linked to crash risk for individual carriers. Consistent with prior research, about 160 of the 754 regulations with data available in this time period had sufficient variation across carriers for analysis. Of the approximately 160 regulations with sufficient violation data, less than 14 were consistently associated with crash risk, across statistical models. These results suggest that the specific weights that SMS assigns to many regulations when calculating safety risk cannot be directly validated with empirical data, and many of the remaining regulations do not have meaningful associations with crash risk at the carrier level.

Data and Methods

We assembled data for a population of motor carriers using the MCMIS snapshot files dated December 2010 and 2012. Specifically, we identified carriers that were actively operating in each of two time periods: from December 2007 through December 2009 (the “pre-period”) and from December 2009 through June 2011 (the “post-period”). We defined an active carrier as one that is as outlined in Appendix I, consistent with FMCSA’s definition of active carriers for its Effectiveness Testing and other analyses. For each of the approximately 315,000 carriers that met these criteria, we extracted data on the number of regulatory violations and crashes incurred in each time period, along with the number of inspections, vehicles, and use of straight versus combo trucks, among other variables, from the crash and inspection tables in MCMIS.

The goal of our analysis was to predict crash risk in the post-period, using data on regulatory violations, crash data, and carrier characteristics measured in the pre-period. We developed a series of linear and generalized linear regression models to predict two measures of crash risk for individual carriers: a binary indicator for having crashed in the post-period and the ratio of crashes to vehicles. Estimating and evaluating all potential models and model types was not the goal of these analyses. Rather, we sought to estimate the associations between regulatory violation rates and crash risk at the carrier level, in order to validate the violations’ severity weights in SMS.

We reduced the list of 754 regulations whose violations are tracked in MCMIS to those that had enough variation across carriers for analysis. After excluding 593 violations that had zero variance or zero counts for more than 99 percent of the analysis carriers, we retained data on the violation of approximately 160 regulations for use in predicting crash risk.

As we discuss in appendix II and the body of this report, crash and violation rates based on small exposure measures, generally resulting from carriers with few vehicles, may be estimated with less precision than rates based on larger exposure measures. To better understand and attempt to overcome these rate estimation issues and assess the sensitivity of our results, we used both ordinary and empirical Bayesian estimators of crash and violation rates. In addition, we estimated separate models limited to carriers that had more than 20 vehicles.

These methodological choices produced 8 groups of models, as described in table 7. The groups were defined by the combined categories of crash measure (binary crash status versus Bayesian crash rate), methods of violation rate estimation (ordinary versus Bayesian), and carrier size (full data or restricted to more than 20 vehicles). These parallel analyses allowed us to assess the sensitivity of our results to different assumptions.

For each of the eight model groups, we include three sets of covariates to predict crash risk in the post-period: “Simple model:” indicator (binary) for crashing in the pre-period, carrier size, and carrier type (percent straight versus combo). “Full model:” predictors in the simple model, plus all violation rates with viable data in the pre-period. “Stepwise full model:” We applied a stepwise selection algorithm applied to all predictors in the “full model,” in order to select the most predictive covariates. The algorithm’s constraints required a p-value of 0.30 for a covariate to enter the model and 0.35 to remain in the model.

To avoid over-fitting our models to any particular sample of data, we divided our data using a random method to form a model-building sample and a validation sample. We used the model-building sample to estimate the models described above and the validation sample to assess the accuracy of the model’s predictions of crash probability against new data. When seeking to develop statistical methods for predictive purposes, this type of out-of-sample validation is extremely useful to ensure that any method identified can consistently predict well on all samples of data, not just the sample that was used to develop the method. This is an important limitation of prior evaluations of SMS, which, to our knowledge, have not used replication samples to avoid over-fitting when identifying predictive violation types or methods of identifying higher-risk carriers.

Model selection required addressing statistical estimation issues, such as instability of the parameter estimates caused by co-linearity of predictors or lack of variability in the predictors, and other model fitting concerns.

For the linear crash rate models, the dependent variable required a log transformation to remove non-constant error variance, which would invalidate results if left untreated. These statistical issues resulted in sub- models within the major model groups that were explored until a stable model resulted. Therefore, the results within each model group focus on three sub models, when applicable: simple, stepwise and full, where stepwise is the model that eliminated independent variables until a stabilized model with estimable coefficients resulted. See table 8 for the final list of 30 models and subsamples.

Evaluation of Models

Models that use the SMS violation information do not fit well according to various measures discussed below. In addition, the violation rates, as measured in SMS, do not have a strong predictive relationship with crashes, regardless of whether the observed or the Bayesian violation rates are used as inputs.

Models for crash status (yes/no) were examined for stability of parameter estimates, fit statistics, number and types of violations that were predictive and that were stable, and future predictive performance according to these measures. Models for Bayesian crash rates were examined for stability of parameter estimates, fit statistics, number and types of violations that were predictive, predictive power and future predictive power. Some of the diagnostics cannot be compared in absolute terms, but rather should be compared across models fit to the same data. For example, the AIC must be compared across competing models fit on the same data.

The crash status (yes/no) model was evaluated in the out-of-sample validation data, where each model was re-fit on the validation sample, and the diagnostics were examined and compared to those from the model-building sample. As an additional sensitivity analysis, the same set of inputs for each of the model groups one through four were also fit using a Bayesian crash rate outcome, via a linear regression fit to the model- building sample. Results were compared.

Model Results

Since diagnostics will differ according to the outcome measure, crash status (yes/no) versus crash rate, information for these outcome types is displayed separately. For results of models for the crash status (yes/no), see tables 9 and 10. For results for the Bayesian crash rates, see table 11. Given that a high value of the H-L p-value (close to 1) indicates good model fit, according to this measure, most of the models fail to fit acceptably, and none of the models fit well.

Within the same data, a lower value of the AIC indicates better fit; therefore, the stepwise models perform best, and do nearly as well regarding the ROC and generalized R-squared when compared to the more complicated full model. But even for the stepwise models, the ROC and R-squared do not indicate a strong predictive relationship. This finding is echoed by the number of effects in the model, relative to the number of potential violations (about 160) and the number of stable effects.

One aspect of predictive power is the ability for a model to discriminate the observed outcomes based on model predictions. Classification tables describe a model’s classification accuracy with correct and incorrect classifications, as measured by sensitivity (correctly predict an event) and specificity (correctly predict a non-event), and false positive (incorrectly predict a non-event) and negative rates (incorrectly predict an event).

Classification tables for the simple, full, and stepwise model within a model group are presented in table 10. The observed proportion of crashes, approximately 0.2 for the unrestricted data and 0.66 for the data restricted to carriers with more than 20 vehicles, is used as the cut-point to classify predicted probabilities for a carrier into a predicted event (crash) versus non-event (no crash). The predicted crash status for a particular model is compared to the actual post-crash status, resulting in a series of table rows, one for each model, that examine the false positives, false negatives, and other quantities that help evaluate the predictive quality of a model.

For unrestricted data, the false negative rate (or the rate that results from incorrectly classifying a carrier to a non-alert status), is relatively low (around 11 percent) compared to the false positive rate (ranges from about 56 to 58 percent). This is a desired result if it is considered more appropriate to be conservative and put a carrier in alert status, even if that alert status is incorrect (false positive), compared to misclassifying a carrier into non-alert when an alert would be called for (false negative). The restricted data have a higher false negative rate (from 42 to 44 percent) than false positive rate (around 14 to 19 percent), and this false negative rate is also higher than the full data false negative rate. For the restricted data with higher false negative rates, this means a higher percentage of carriers are being classified in non-alert when they have crashed than the percent classified as alert, but that did not crash, and such a scenario is not desirable under a conservative preference toward low false negative rates. In addition, the sensitivity and specificity are both moderate at best within data (restricted versus full), further evidence of the inability for models to discriminate.

To address whether crash status (yes/no) has a different relationship with violations than the crash rate, we compare conclusions of crash status (yes/no) versus crash rate models. Examining sensitivity to the prediction of crash status (yes/no) versus crash rate, the stepwise selected model will be compared to logistic regression results for the model-building and the validation sample (see Table 11).model indicates that the numbers of effects that are related to crash rate are small, and that the better fitting models tend to have only a few predictors included. Specifically, Mallow’s Cp statistic indicates a model is preferable when Cp is around or smaller than the number of effects (p), and the model is more parsimonious than competing models. The model fit to the restricted data, where carriers have greater than 20 vehicles, (stepwise model number 22), includes only 34 stable effects, and 72 effects altogether, but the model fit is more stable (i.e., relatively fewer unstable effects) and has the best (lowest) Cp, while also having similar explained variance and low AIC. However, it is interesting to note that the simple model, model 21, performs similarly according to some measures, such as Root MSE and R-squared, though this model does not contain violation rate information.

Model Predictive Power

Comparing how well the models perform when applied to the validation sample that consists of new observations——which are not included in the model-building sample—informs the precision of SMS with respect to predicting crashes. We examine the number of violations and the violation types that are included across the model groups (logistic and linear) and sub-models (stepwise and full). We compare this to the number of models within which each violation was found to be a significant and a stable predictor of crash outcomes. Importantly, of the reduced set of approximately 160 violations considered, only 13 violations were significant in at least half of the 24 models that incorporate violations (i.e., stepwise and full models).

There were 10 different possible models for the logistic model-building sample, and these were also evaluated on the validation sample and on the model-building sample, but with a linear regression setting, resulting in 30 possible models. However, we regarded only 24 of these 30 models as informative since we exclude the 6 simple models that ignore the pre- violation information. Of the violations considered, only speeding (violation 3922S) and failure to use a seatbelt while operating CMV (39216) were significant and stable in all 24 models. A similar picture arises for some other violations, though many of the models did not result in a significant relationship between the violation in question and the crash outcome, as indicated in table 12. Only 41 violations were significant in 5 or more models out of 24. However, even for the top 13 violations with respect to frequency of significance and stability across the 24 models, predictive power is still affected by poor model diagnostics. This is echoed in the results from the predictive relationship when compared to the linear regression model for Bayesian crash rates (results in table 11), where the model that excluded all violations performed similarly to models that included some significant violations. Whether modeling crash status (yes/no) or a crash rate, the predictive power of SMS violations is weak.

When comparing the predictive power of the models that result from the model-building sample, once applied to the validation sample, there is a consistent picture regarding the model fit (see table 13). In particular, the model fit is generally poor according to the H-L value; the stepwise model tends to perform better according to the AIC, but the ROC, adjusted R2, and percent discordant do not indicate the models have a strong ability to discriminate and predict future crashes. Classification tables that result from evaluating the model-building sample models, but estimated from the validation sample, generally resulted in similar results to those presented in table 10.

Conclusions

The predictive power observed in these modeling and sensitivity analyses indicates that SMS may be less precise than what is reported and that the available information on violations is limited for the purpose of scoring carriers or predicting their crash risk. Regardless of which type of model we fit, we see that the predictive power of our models is low, and the use of the SMS violations in predicting future crashes is not very precise. The number of stable and significant effects across the various model-fitting scenarios that include violations is small. For the about 800 violations in SMS, only around 160 met the basic criteria of non-zero variance and non-zero counts for at least 1 percent of the sample. Of these, only two violations (speeding and failure to wear a seatbelt while operating a CMV) consistently appeared as a stable predictor of crashes, regardless of data and model. While some other violations appeared in models, only 13 were significant and stable in at least half of the models, most were significant in no more than half the models examined, and most often in fewer than 5 of the models. The results did not vary substantially according to whether observed versus Bayesian violation rates, crash versus Bayesian crash rates, or restricted data (carriers with more than 20 vehicles) versus full data were used to estimate crashes. Therefore the modeling attempts did not overcome the issues that result from small exposures. The results were generally confirmed when evaluated on a validation sample, indicating the future prediction is stable, yet not strong. Ultimately, much of the variance in crash predictions remains unexplained, regardless of the model and model-building data, so that the SMS might be less precise when the objective is to predict crashes.

Appendix VI: Descriptive Statistics on Motor Carrier Population and Results of GAO’s Analysis

This appendix provides additional information and illustrations of the distribution of motor carrier population included in our analysis such as carrier size, number of crashes, inspections, and high risk status (see table 14). It also provides results of our analysis on the number and percentage of carriers above or below intervention thresholds, as well as the frequency and rate of crashes for each of those groups of carriers within each BASIC using FMCSA’s methodology and the illustrative alternative methodology (i.e., using a stronger data sufficiency standard) demonstrated earlier in the report. In addition, this appendix provides summary statistics of the various motor carrier populations used in FMCSA and GAO analysis. These statistics include, among other things, the numbers of carriers with an SMS score (i.e., “measure”) and the number of carriers above an intervention threshold in at least one BASIC. Finally, this appendix provides the complete graphical results of our analysis of FMCSA’s violation rates, safety event groups, and distribution of SMS scores for carriers above FMCSA’s intervention threshold using FMCSA’s methodology.

Table 15 contains the results of our analysis using FMCSA’s SMS 3.0 methodology. This analysis calculated the number and percentage of carriers above and below intervention thresholds for each BASIC using carrier data from December 2007 through December 2009, and determined which carriers subsequently crashed during the 18-month evaluation period, December 2009 through June 2011. The analysis also presents aggregate crash rates for comparison purposes.

Table 16 contains the results of our analysis using an illustrative alternative incorporating a stronger data sufficiency standard, among other things, as described elsewhere in this report (e.g. carriers with 20 or more inspections or 20 or more vehicles, depending upon the BASIC). As in the previous table, this analysis calculated the number of carriers above and below intervention thresholds for each BASIC using carrier data from December 2007 through December 2009, and determined which carriers subsequently crashed during the subsequent 18-month period, December 2009 through June 2011. The analysis also presents aggregate crash rates for comparison purposes.

Table 17 contains selected SMS outcomes based on results reported by FMCSA’s and from GAO’s analysis.

The following figures are graphical results of our analysis of the average and range of violation rates for carriers, percentage of carriers above FMCSA’s intervention thresholds for various safety event group categories, and distribution of SMS scores for carriers above FMCSA’s intervention thresholds using FMCSA’s methodology as discussed in the body of this report above. Figures 10 through 16 contain the average and range of violation rates for all carriers (where a violation rate could be calculated) by carrier size, for all the BASICS. Figures 17 through 25 contain the percentage of carriers above intervention thresholds within safety event groups for each BASIC. Finally, figures 26 through 32 show the distribution of carriers above intervention thresholds for each BASIC by carrier size.

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, H. Brandon Haller, Assistant Director, Russell Burnett, Melinda Cordero, Jennifer DuBord, Colin Fallon, David Hooper, Matthew LaTour, Grant Mallie, Jeff Tessin, Sonya Vartivarian, and Joshua Ormond made key contributions to this report.