Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions
Released: 2016-10-27T00:00:00

Letter

Background

FMCSA was established within the Department of Transportation (DOT) in January 2000, and is tasked with promoting safe commercial motor vehicle operations and reducing large truck and bus crashes, injuries, and fatalities. It seeks to achieve this reduction through regulation, enforcement, and partnerships with stakeholders, among other activities, and with full accountability to the public through transparency, results- oriented performance measuring, and managing for results. Since fiscal year 2010, FMCSA’s total budget authority to conduct these activities has remained relatively stable, increasing about 6.5 percent from fiscal year 2010 through fiscal year 2016 (see table 1). Funding for implementing and applying interventions is included in both the Motor Carrier Safety Operations and Programs and Safety Grants budget authorities.

The vast majority of FMCSA’s staff are located in field offices, including divisions and service centers. Field staff are primarily responsible for implementing FMCSA’s compliance and enforcement activities, including investigations. Federal Safety Investigators represent the majority of FMCSA’s compliance and enforcement program staff (see table 2). In addition, FMCSA partners with state agencies to perform some intervention activities, such as conducting carrier investigations; however, FMCSA is responsible for ensuring that commercial motor carriers under its authority comply with federal safety regulations. FMCSA’s Office of Enforcement is the primary office responsible for developing policy for FMCSA’s compliance and enforcement program, and overseeing the implementation of intervention activities.

The CSA Program

The CSA program is intended to improve the effectiveness of FMCSA’s compliance and enforcement programs, while more efficiently using its resources to reach carriers that pose the highest safety risk. The CSA program has three key components: (1) the Safety Measurement System (SMS), meant to identify high-risk carriers by using data from roadside inspections and crashes; (2) interventions, which are intended to help carriers address safety problems; and (3) the Safety Fitness Determination rule. In contrast to the one investigation intervention, the compliance review, available under FMCSA’s previous approach, FMCSA expects the CSA program to more effectively change unsafe behaviors by reaching and intervening with more potentially unsafe carriers earlier using a range of intervention types to enforce compliance with safety regulations. In 2014, we reported on the effectiveness of the SMS component of the CSA program.

FMCSA conducted an Operational Model Test of the program from February 2008 through June 2010 in nine pilot states. In December 2010, FMCSA began implementing the CSA program in three phases: (1) implementation of SMS and some intervention types nationwide, (2) introduction of new investigative techniques and the Safety Management Cycle, and (3) nationwide rollout of all intervention types and FMCSA’s new investigative software, the Safety Enforcement Tracking and Investigation System (SENTRI).

After a series of four commercial motor vehicle crashes—two involving buses and two involving trucks—that together resulted in 25 deaths and injuries to 83 people, the National Transportation Safety Board investigated and, in November 2013, made recommendations to improve the quality of FMCSA’s compliance review processes. The Secretary of Transportation tasked the Federal Aviation Administration, as a peer of FMCSA within DOT, to conduct a review and develop appropriate recommendations for DOT’s response to the National Transportation Safety Board. The Federal Aviation Administration formed an Independent Review Team, which in July 2014, issued a report that included a range of recommendations intended to support both incremental and transformative improvements to FMCSA’s compliance and enforcement programs. We discuss some steps FMCSA is taking to address these recommendations later in this report.

CSA Interventions

Under the CSA program, FMCSA can select from a range of eight intervention types, intended to give FMCSA the flexibility to address motor carriers’ specific safety problems. Four of the intervention types were newly introduced under the CSA program; FMCSA had been applying the other four intervention types prior to the program (see table 3). Each type falls into one of the following intervention categories: early contact, investigation, and follow-on interventions.

Before the CSA program, FMCSA used one investigation intervention type—the onsite compliance review—and three follow-on intervention types. Compliance reviews required investigators to examine every part of a carrier’s operations and were thus extremely resource intensive to conduct. As a result, FMCSA and its state partners investigated only about 3 percent of active carriers. According to FMCSA Motor Carrier Safety Progress Reports, federal or state investigators conducted between approximately 15,000 and 17,000 compliance reviews each year from fiscal year 2006 through fiscal year 2009.

CSA Intervention Process

Under the CSA program, FMCSA has established a process for measuring the relative safety risk of carriers in seven Behavioral Analysis and Safety Improvement Categories (BASICs), prioritizing carriers based on risk, assigning an appropriate intervention type, and investigating or enforcing compliance with regulations (see fig. 1). The CSA intervention process has no set progression, and based on existing guidance, FMCSA applies one or more interventions depending on the circumstances of each case. For example, FMCSA may directly assign an onsite comprehensive investigation to a carrier without first assigning another type of intervention. FMCSA may also apply multiple interventions to a carrier over time, and as a result, common patterns in FMCSA’s application of CSA interventions can be identified. For example, if a carrier receives a warning letter as a first intervention and its safety performance does not improve, FMCSA may assign the carrier a second intervention, such as an onsite focused investigation. If FMCSA identifies violations during an investigation, such as the onsite focused investigation, that warrant enforcement, FMCSA may assign a third intervention, such as a notice of claim. In such a case, the resulting intervention pattern would be: (warning letter) → (onsite focused investigation) → (notice of claim).

Every month, FMCSA uses SMS to generate percentiles in any of seven BASICs for carriers with sufficient data. However, as we reported in 2014, the SMS methodology contains limitations that reduce FMCSA’s ability to reliably assess carriers’ safety risks because FMCSA lacks sufficient safety performance information on the majority of carriers. We identified the lack of sufficient information as a particular limitation for carriers with few inspections and vehicles because their underlying violation rates can have artificially low or high values, or greater variability, which affects the precision of the BASIC percentiles used by FMCSA in comparing carriers to one another.

Nonetheless, FMCSA uses these BASIC percentiles to identify potentially risky carriers, to prioritize them for intervention, and to automatically generate warning letters. FMCSA automatically prioritizes carriers into risk-based categories of escalating urgency based on the extent to which a carrier’s BASIC percentiles exceed certain combinations of designated thresholds, in addition to the carrier’s intervention history and unresolved violations. For example, under the new high-risk definition FMCSA adopted in March 2016, for a carrier to be considered high-risk, its BASIC percentile has to be above 90 in at least two of four BASICs—unsafe driving, crash indicator, hours-of-service compliance, or vehicle maintenance—and the carrier cannot have had an onsite comprehensive investigation in the last 18 months. In addition to changing the high-risk definition, FMCSA implemented a new prioritization approach with five risk-based categories, from most to least urgent: high-risk, moderate-risk, risk, warning letter, and monitor. According to FMCSA, high-risk carriers are the agency’s highest investigative priority.

After FMCSA prioritizes carriers for intervention, FMCSA division managers decide which intervention type carriers should receive based on priority level, guidance, and carrier history among other things. The principal guidance document for assigning intervention types is the electronic Field Operations Training Manual (eFOTM). Although eFOTM includes some requirements, division managers have some discretion to select investigation and follow-on intervention types based on additional information, such as complaints received, significant crashes, and professional judgment. For example, FMCSA must investigate non- frivolous written complaints that allege substantial violations regardless of whether the carrier is prioritized for intervention, but have discretion to determine the most appropriate investigation type based on the safety problems associated with the complaint, the carrier’s BASIC percentiles, and the carrier’s history. However, for carriers identified as high-risk, FMCSA must conduct an onsite focused or onsite comprehensive investigation.

Federal Safety Investigators and their state partners follow eFOTM guidance when conducting investigation and follow-on interventions. In April 2013, FMCSA introduced enhanced investigative techniques (EIT) that are intended to help investigators identify the root cause of a motor carrier’s safety problems. While financial and legal penalties are typically applied following an investigation, FMCSA may levy financial penalties against a carrier without an investigation if it believes there is sufficient evidence, such as evidence that the carrier operated after being placed out of service. Investigators may also request a change to the intervention type for some carriers when they find new and pertinent information that was not available at the time of the assignment.

FMCSA Information Systems

FMCSA’s information systems are critical to its data-driven enforcement and compliance programs and are intended to provide real-time access to data for the enforcement community, the transportation industry, stakeholders, and the general public. Field staff input intervention data through a variety of field information systems. These systems are operated on laptop computers in the field. For example, field staff use the Compliance Analysis and Performance Review Information (CAPRI) system to enter investigation intervention data, such as investigatory files and safety violations identified. Similarly, they use CaseRite to enter legal enforcement information. As previously discussed, FMCSA plans to introduce a new field information system called SENTRI, which will consolidate the legacy information systems that field staff use to upload information related to interventions.

Once uploaded by field staff, intervention data are stored and analyzed on multiple central information systems. FMCSA staff may access CSA intervention data on these systems through a centralized portal and use the data to monitor carriers’ safety performance, among other things. For example, the Motor Carrier Management Information System (MCMIS) includes motor carrier performance data including inspection and investigation results, enforcement data, and state-reported crashes. FMCSA also uses the Enforcement Management Information System (EMIS) to monitor, track, and store data related to FMCSA enforcement actions, including follow-on interventions. FMCSA’s Analysis and Information Online system provides public access to descriptive statistics and analyses regarding commercial vehicle, driver, and carrier safety information.

Information Technology Delays Have Hampered Full Implementation of CSA Interventions and Limited the Quality of Available Information

FMCSA Has Delayed Implementing Two New CSA Intervention Types Nationwide Until It Develops Information Technology Software

Although FMCSA implemented all four new CSA intervention types in pilot test states, the agency chose to delay implementing two of the new intervention types in the remaining states until it develops information technology (IT) software.

Implementation in Pilot Test States: FMCSA implemented the entire range of CSA interventions—including all four new CSA intervention types—in nine pilot test states as part of the Operational Model Test that FMCSA conducted from February 2008 through June 2010. The test was intended to help the agency assess the four new intervention types and identify any features that needed to be adjusted prior to implementing them nationwide, among other things. According to FMCSA headquarters officials, personnel experienced challenges using multiple legacy information systems that were not designed to support FMCSA’s application of the expanded range of interventions under the CSA program. For example, FMCSA’s data analysts found it difficult to extract data from information systems needed to monitor and oversee the agency’s application of interventions.

Implementation in Non-Pilot Test States: According to headquarters officials, in July 2010 FMCSA chose to delay implementing two of the four new CSA intervention types—offsite investigations and cooperative safety plans—in the remaining non-pilot test states until it completes its development of SENTRI software. However, FMCSA decided to implement the remaining two new intervention types—warning letters and onsite focused investigations—nationwide because they believed those two interventions were demonstrated as effective during the Operational Model Test and that delays would hinder safety benefits for the public (see fig. 2). The Operational Model Test evaluation found that offsite investigations demonstrated a similar pattern of effectiveness as onsite focused and onsite comprehensive investigations.

FMCSA headquarters officials told us that developing and implementing SENTRI is important to help field staff and their state partners conduct their work. Field staff currently may use a variety of legacy information systems to apply and manage interventions. Principally, field staff use the CAPRI system to prepare for investigation interventions and to report their results. However, CAPRI was designed to support traditional compliance review investigations, not the expanded range of investigation types under the CSA program. According to FMCSA officials, this has resulted in field staff taking time-consuming additional steps to report their application of interventions. For example, some division administrators spent additional time reviewing how investigators entered information into CAPRI to determine the correct investigation type performed. According to officials, SENTRI is expected to help address these inefficiencies by consolidating investigative, follow-on, reporting, and other functions into a single interface. FMCSA officials also expect SENTRI to improve data consistency and enable better policy and program decisions through improved data quality.

However, FMCSA has faced longstanding delays in developing SENTRI software as part of its broader IT modernization effort. In September 2005, FMCSA initiated a comprehensive overhaul of the way it collects, manages, and conveys safety information. The agency-wide modernization effort was intended to help FMCSA achieve its effectiveness and efficiency outcomes for the CSA program by centralizing FMCSA data and simplifying information access, among other things. According to FMCSA headquarters officials, FMCSA began obligating funds to develop SENTRI software in fiscal year 2009, when it established the business case for the system (see app. II). Since that time, we and the DOT’s Office of Inspector General have reported continuing project delays.

FMCSA hired consultants to identify the causes of, among other things, its IT project delays and actions to remediate them. The resulting March 2013 report found a variety of underlying program challenges. For example, it found that ineffective IT governance practices provided limited visibility into the health of individual projects, contributing to project delays. It also found that the lack of an appropriately scoped and measurable strategy made it unclear whether current resources were effectively prioritized—a challenge that was compounded when priorities shifted on multiple occasions over time.

Officials stated that FMCSA executed a contract in January 2016 to complete the agency’s IT modernization effort. As part of this effort, FMCSA plans to complete its development of SENTRI by April 2017.

FMCSA’s Application of CSA Interventions Has Declined, but Data Limitations Affect the Quality of Available Information

Declining Intervention Application

FMCSA’s application of interventions declined from fiscal year 2012 through fiscal year 2015, according to estimates provided by the agency. FMCSA implemented warning letters nationwide in fiscal year 2011, which resulted in a temporary spike in interventions. However, after this temporary increase, the number of interventions FMCSA applied was less in fiscal year 2015 than in fiscal year 2012 for each intervention type, with notable decreases in offsite investigations (73 percent) and notices of violation (71 percent). In addition, according to FMCSA’s estimates, about 26 percent fewer total investigation interventions were conducted in fiscal year 2015 compared to fiscal year 2012. See table 4 for detailed estimates of FMCSA’s application of interventions. Reasons for these notable decreases are discussed below.

Offsite investigations: Division officials from each of the four pilot test states we interviewed told us they selected offsite investigations less frequently in more recent years, because the number of carriers that met eFOTM criteria for receiving them decreased over time. For example, officials from one division told us their use of offsite investigations decreased because motor carriers’ BASIC percentiles were typically too high or involved too many BASICs to qualify to receive an offsite investigation according to current eFOTM criteria. FMCSA headquarters officials told us they are focused on increasing the use of offsite investigations, because they believe offsite investigations were demonstrated as both efficient and effective. For example, in March 2016 FMCSA established a working group to explore modifying policy to give division managers more discretion in assigning offsite investigations.

Notices of violation: Division officials from four of the eight divisions we interviewed told us they selected notices of violation infrequently because the intervention type was time-intensive to process compared to other intervention types or was not appropriate to address severe safety violations. FMCSA headquarters officials told us that investigators prefer to issue notices of claim instead, because they result in penalties to motor carriers. However, officials stated that investigators may not be aware of other associated FMCSA activities that increase the overall resources used to issue notices of claim. For example, notices of claim require additional legal oversight, which generally requires more resources. Headquarters officials said investigators may choose to issue notices of violation rather than notices of claim, when appropriate, if they better understood this context.

Total investigation interventions: FMCSA headquarters officials told us that total investigation interventions declined because investigators spent more time conducting in-depth reviews of motor carriers’ safety management practices to identify the root causes of underlying safety problems as part of FMCSA’s EIT initiative. FMCSA implemented EIT in fiscal year 2013 as part of continuous improvement efforts and in response to Independent Review Team recommendations. According to officials, using the more time-intensive EIT approach decreased the number of investigations that FMCSA could conduct, particularly since 2012. FMCSA officials stated investigation counts have increased somewhat as investigators adjusted to the new EIT procedures. However, the officials did not expect investigation counts to return to previous levels without additional personnel.

Data Quality Limitations

As FMCSA introduced an expanded range of intervention types under the CSA program, FMCSA did not redesign CAPRI or other legacy information systems to reflect these changes. For example, FMCSA did not redesign CAPRI to include a dedicated data element that would uniquely record the occurrence of each intervention type. Instead FMCSA developed algorithms—rules that can be applied by computer programs—that attempted to reconstruct the occurrence of each intervention type by identifying specific combinations of multiple data elements.

Using legacy information systems for purposes for which they were not designed produced two main limitations that affected the accuracy of FMCSA intervention counts: Data recording limitations: FMCSA headquarters officials stated that the accuracy of CSA intervention counts depended in part upon how users recorded intervention data into FMCSA’s IT systems. For example, although all but 10 states did not implement offsite investigations, the CAPRI system nonetheless allowed investigators in these states to select “offsite” for the “review location” data element. According to FMCSA officials, this inflated counts when FMCSA used “review location” as one of multiple data elements to identify offsite investigations. Similarly, investigators may conduct onsite focused investigations on carriers that receive complaints, but CAPRI requires investigators to select either “complaint” or “focused CR” for the “review reason” data element. Because FMCSA’s algorithm used “review reason” as one of multiple data elements to count onsite focused investigations, this deflated onsite focused investigation counts when investigators selected “complaint” to conduct them.

Evaluation Limitations: FMCSA officials told us they occasionally modified algorithms used to identify the occurrence of intervention types, but generally did not evaluate how the modifications affected the accuracy of intervention counts. According to officials, they modified algorithms for a variety of reasons, such as when the agency changed how it recorded intervention data. Once modified, FMCSA applied the most recent algorithm to all previous data—including historical data. For example, in January 2016, FMCSA removed a redundant data element from the algorithm used to count investigation interventions, a step that changed historical intervention counts. FMCSA officials told us they did not know the extent to which applying the modified algorithm to previous data affected the accuracy of historical counts, because they generally did not evaluate the modification’s effect on count accuracy before applying it. Although the extent of the effect is unknown, even small differences could limit the ability of FMCSA managers to accurately and effectively monitor trends in FMCSA’s application of CSA interventions over time.

FMCSA headquarters officials told us that SENTRI is intended to address the underlying IT challenges that limit the accuracy of CSA intervention counts by creating a dedicated data element that uniquely records the occurrence of each intervention type. Developing SENTRI in a timely manner is particularly critical, because data-driven targeted enforcement is FMCSA’s primary strategy for meeting its safety goals and further delays represent missed opportunities for FMCSA to accurately monitor and improve the CSA program. Moreover, unresolved data limitations would continue to preclude outside entities, such as auditing entities, from assessing the integrity of agency information, including the completeness and accuracy of computer-generated counts.

In May 2016, we initiated a review to determine the extent to which FMCSA has evaluated the effectiveness of selected IT systems, and to assess the extent to which FMCSA has implemented an IT governance structure and plan to complete this work by June 2017. In addition, as we discussed above, FMCSA currently estimates that it will complete SENTRI development in April 2017. In light of our and FMCSA’s ongoing work in this area, we are not making a recommendation on this matter in this report.

FMCSA Evaluated the Effectiveness and Efficiency of CSA Interventions, but the Evaluations Had Limitations

In its Strategic Plan: Fiscal Years 2015–2018, FMCSA identified the improved effectiveness and efficiency of CSA interventions as strategic outcomes. FMCSA has evaluated both of these strategic outcomes, but its evaluations had limitations. Specifically, FMCSA’s effectiveness evaluations did not produce sufficiently complete, appropriate, and accurate information on individual intervention types because of design and methodological limitations. Additionally, FMCSA’s efficiency evaluation is no longer current, because FMCSA has not taken steps to update the evaluation’s cost estimates, despite changes in the time and resources required to conduct CSA interventions.

FMCSA’s Effectiveness Evaluations Did Not Produce Sufficiently High- Quality Information

FMCSA’s Strategic Plan: Fiscal Years 2015–2018 identifies improved effectiveness as a strategic outcome for CSA interventions. According to FMCSA, the agency conducts regular evaluations to determine how effectively its programs are achieving their effectiveness and other intended outcomes. To evaluate the effectiveness of CSA interventions specifically, FMCSA developed a statistical model intended to annually evaluate the combined effects of all of its interventions. According to FMCSA, the model is a revised version of a prior model that FMCSA used to evaluate the effectiveness of compliance reviews, before the implementation of the CSA program added new intervention types. FMCSA has also evaluated intervention effectiveness in other studies, but according to officials, the new annual model is the agency’s primary method of assessing intervention effectiveness.

In a January 2015 report, the annual effectiveness model estimated the combined effect of four CSA intervention types on the crash rates of carriers in four size groups from fiscal year 2009 through fiscal year 2011. To assess effectiveness, the model estimated the change in group crash rates before and after carriers received one or more interventions, compared to the change in a comparison group of carriers that did not receive an intervention. This design accounted for the effects of some external factors that also could have influenced group crash rates, such as broad changes in weather or economic conditions. When used appropriately, a comparison group design is a key strength of a model, such as the one used by FMCSA, as is the use of statistical inference to evaluate the certainty of the model’s results.

Federal standards for internal control state that agencies should use quality information to determine the extent to which they are achieving their intended program outcomes. Characteristics of quality information include complete, appropriate, and accurate information that helps management make informed decisions and evaluate the entity’s performance in achieving strategic outcomes. Because, according to headquarters officials, the annual effectiveness model is the primary method FMCSA uses to evaluate CSA intervention effectiveness and FMCSA intends to use the model on a recurring basis, we conducted a detailed assessment of the model. As discussed below, we identified several design and methodological limitations in FMCSA’s annual effectiveness model, including the design of its comparison groups, a design limitation that can impact the quality of information that the model produces (see app. III for our complete assessment).

Information Completeness

FMCSA’s report concluded that applying at least one intervention reduced crash rates for three out of four carrier size groups in fiscal year 2009 and fiscal year 2011 and provided positive safety benefits. The report also concluded that warning letters independently reduced group crash rates and increased safety benefits. However, FMCSA’s annual evaluation did not provide sufficiently complete information on intervention effectiveness because the evaluation did not assess all intervention types or intervention patterns that FMCSA commonly applies. Specifically, the evaluation did not explicitly measure follow-on interventions, including cooperative safety plans, notices of claim, and notices of violation. According to the January 2015 report, the evaluation did not include cooperative safety plans because MCMIS data for that intervention were not consistently complete or accurate. FMCSA officials told us that the model does not specifically exclude notices of claim and notices of violation, but rather simply does not distinguish between investigations that result in follow-on interventions and those that do not. According to FMCSA, the model does not make this distinction because FMCSA typically applies follow-on interventions, such as notices of claim, within 90 days of conducting an investigation, based on the investigation’s findings.

We determined that FMCSA’s model cannot analyze such intervention patterns, because the agency designed it to identify only a carriers’ first intervention in a fiscal year. As a result, FMCSA does not have information on the unique effectiveness of its specific follow-on interventions, which are critical for enforcing regulatory compliance. FMCSA could potentially increase the breadth of analysis for its model by using a design similar to the evaluation of FMCSA’s Operational Model Test, conducted by UMTRI in August 2011. That evaluation identified common patterns of interventions with sufficient data for analysis and then estimated each pattern’s effectiveness. The combined estimates supplemented those for individual intervention types.

Information Appropriateness

The annual evaluation does not provide information that appropriately reflects how FMCSA designed and implemented CSA interventions. The agency designed CSA interventions to replace the resource-intensive “one-size-fits-all” compliance review with a range of intervention types intended to better address safety problems specific to individual carriers. According to FMCSA’s Strategic Plan: Fiscal Years 2015–2018, the agency’s strategy to reduce the number of unsafe and high-risk carrier behaviors is to create and apply appropriate interventions. As we previously discussed, FMCSA’s application of individual intervention types depends upon a combination of eFOTM rules and managers’ day- to-day discretion. For example, according to eFOTM, managers generally assign offsite investigations when a carrier has percentiles above intervention thresholds in three or fewer BASICs. However, managers have the discretion to apply an onsite focused investigation instead, based on the carrier’s circumstances. Additionally, FMCSA may apply multiple intervention types for the same carrier over time, resulting in commonly observed intervention patterns. For example, one official told us that most carriers that FMCSA investigates have, at one time, been issued a warning letter.

In contrast to the design and implementation of CSA interventions, FMCSA’s model does not include an assessment of either individual intervention types or common intervention patterns. Instead, the model estimates the impact of all interventions combined that were performed during a 12-month period being measured. Recommended practices for program evaluation recommend that program managers attempt to separately evaluate multiple types of program activities that seek to achieve a common outcome–in this case, multiple intervention types that seek to improve carrier safety performance. Consistent with these practices, FMCSA’s Strategic Plan: Fiscal Years 2015–2018 calls for the agency to use data to make smarter day-to-day decisions and to determine the impact that various rules have on decreasing crashes, injuries, and fatalities by conducting regular program evaluations and effectiveness reviews.

Therefore, to provide information appropriate to the design and implementation of CSA interventions, an evaluation should assess how effectively each intervention type or common intervention patterns addressed the safety problems of the carriers that received them. This specific information could help FMCSA identify the circumstances under which different types of interventions are effective and help managers optimize their choice of interventions on a day-to-day basis as the agency implements the program.

Although officials told us that FMCSA designed the model to measure the cumulative effect of FMCSA contact with carriers through CSA interventions and not to analyze individual intervention types, FMCSA used the model to draw conclusions about the safety benefits of warning letters, one of the most common intervention types used according to FMCSA estimates. Specifically, FMCSA concluded that “this suggests that the warning letter in and of itself can be an effective tool for improving motor carrier safety.” However, as accepted practices for designing evaluations explain, quality evaluations should draw conclusions commensurate with the power of the design. Because FMCSA did not evaluate the separate effect of warning letters, it lacks specific analytical evidence to support its conclusion. As a result, FMCSA lacks quality information needed to estimate how each intervention type, including warning letters, or common intervention patterns affect motor carrier safety performance and address carriers’ specific safety problems.

FMCSA headquarters officials stated that they were unsure if it was possible to measure the effects of individual intervention types or common intervention patterns because the quantity of available data may not be sufficient to produce reliable results. However, as previously discussed, FMCSA’s August 2011 evaluation of the CSA Operational Model Test, conducted by UMTRI, assessed the effectiveness of each intervention type and common intervention patterns for carriers that received multiple interventions. An evaluation of these effects was possible for the Operational Model Test, despite the fact that UMTRI had less data available than FMCSA would generate after nationwide implementation of the CSA program. According to the study, UMTRI assessed such patterns to provide a more detailed look at the effectiveness of the interventions, in light of how FMCSA actually applied them in the field. Furthermore, in March 2016, FMCSA conducted separate evaluations of how two specific intervention types—onsite focused and offsite investigations—influenced carriers’ BASIC percentiles in calendar years 2011 and 2012. By combining multiple years of intervention data, as FMCSA did in these evaluations, rather than using a fiscal year construction, which officials told us FMCSA used in the annual evaluation for administrative reasons, FMCSA could potentially overcome data sufficiency limitations (see app. III).

FMCSA and independent evaluators have identified a need for this level of detailed information in the management and implementation of interventions. For example, an internal FMCSA working group determined that the agency needed a more detailed understanding of the effectiveness of onsite focused investigations to empower investigators to select the most appropriate intervention to change carrier behavior. Additionally, in its 2014 assessment, the Independent Review Team that DOT tasked with reviewing FMCSA’s compliance review processes identified a need for FMCSA to perform consistent, detailed, evaluations of effectiveness by enforcement tool, such as intervention types. Without detailed evaluations, the team said that FMCSA would be unable to focus resources on using its most effective tools or to reconfigure tools that are not meeting the agency’s goals.

Nonetheless, as discussed earlier in this report, FMCSA’s ability to accurately identify specific intervention types is limited. Accepted practices for designing evaluations state that quality evaluations should rely on credible data that are sufficiently free of errors that could lead to inaccurate conclusions. Taking steps to reliably and accurately identify each intervention type in the data used to support its evaluations would help FMCSA conduct evaluations that produce information appropriate to the design and implementation of CSA interventions.

Information Accuracy

Uses and Characteristics of Comparison Groups A comparison group, in the context of typical designs for evaluating program effectiveness, represents what would have happened in the absence of a program and is used to rule out alternative explanations for changes in outcomes. In a truly randomized experiment, this w ould be the control group. In a quasi- experimental evaluation, like FMCSA’s annual effectiveness evaluation, w here participants (i.e., carriers) are not sorted randomly into groups, the comparison group should be constructed to be as similar as possible to the group being influenced by the program (i.e., carriers receiving interventions), in order to draw strong conclusions about the effects of the program. The groups should be similar enough that any difference in outcome can be plausibly attributed to the intervention type being evaluated.

FMCSA’s annual and separate effectiveness evaluations had methodological limitations that limited their ability to accurately attribute changes in carrier safety behavior solely to interventions. Because of these limitations, FMCSA may not have accurately accounted for factors other than FMCSA’s interventions that could be responsible for the outcomes observed. Most notably, FMCSA did not consistently use a comparison group design, which compares outcomes among carriers that did and did not receive interventions, for its effectiveness evaluations. When FMCSA did use this design, it constructed comparison groups that did not sufficiently account for external factors that could affect group crash rates. According to recommended practices for designing program evaluations, comparison group designs are typical for assessing program effectiveness, because they can isolate a program’s unique effects when the comparison groups are sufficiently similar to the groups affected by a program. See appendix III for a complete assessment of the limitations we identified in FMCSA’s annual effectiveness evaluation, along with accepted practices that could help FMCSA to address them.

FMCSA’s separate evaluations of onsite focused and offsite investigation effectiveness did not use a comparison group and so, as previously discussed, did not account for external factors that could have influenced changes in motor carriers’ BASIC percentiles. Officials stated that they determined that a comparison group method was not appropriate for FMCSA’s evaluation of onsite focused investigations because the officials were concerned that a comparison group would overstate the effectiveness of onsite focused investigations due to the differing safety profiles of carriers that receive each intervention type. FMCSA headquarters officials stated that they chose to use the same methodology as the onsite focused investigation effectiveness evaluation for the separate evaluation of offsite investigation interventions.

Although FMCSA did use a comparison group approach in its annual effectiveness evaluation, we identified limitations with FMCSA’s approach that affect the model’s ability to accurately attribute changes in crash rates to interventions. For example, FMCSA’s comparison group was observed over a different measurement period from the carriers that received interventions, so that the two groups were not perfectly matched on broad changes, such as weather and economic changes. By not matching the measurement period, FMCSA’s use of a comparison group was limited in its ability to account for external factors, as intended.

Additionally, in the 2015 evaluation, FMCSA’s comparison groups held constant the effects of carrier size, but did not hold constant key factors that could influence intervention outcomes, such as pre-intervention safety behaviors as measured by regulatory violations or BASIC percentiles. FMCSA headquarters officials said that comparison groups in the model accounted only for carrier size because of data limitations and their concern that accounting for too many additional variables would reduce the power of the model. Specifically, officials said that FMCSA does not currently have sufficient data to assess the effectiveness of agency interventions for motor carriers with different characteristics, such as the total number of miles a carrier’s vehicles travel per year. However, FMCSA has previously used the data it has available to hold constant important factors external to the program and attribute changes in outcomes to its interventions with greater accuracy (see app. III). For example, FMCSA’s August 2011 evaluation of the Operational Model Test, conducted by UMTRI, matched groups of carriers that did and did not receive various types of interventions on several key characteristics, such as the distributions of pre-intervention crash rates and BASIC percentiles. Without using a robust comparison group or a similar method, FMCSA cannot accurately determine whether changes in motor carrier safety performance are a result of interventions or whether other factors are responsible.

Without more complete, appropriate, and accurate information on the effectiveness of individual CSA intervention types, FMCSA lacks information it needs to make informed decisions and evaluate its performance in achieving its effectiveness outcome for CSA interventions. Additionally, FMCSA’s ability to accurately identify specific intervention types by analyzing MCMIS and EMIS data is limited due to the data limitations which we described earlier in this report. Without taking steps to address these limitations, FMCSA cannot accurately evaluate how effectively CSA interventions improve motor carriers’ safety performance.

FMCSA’s Evaluation of Intervention Efficiency Is Based on Outdated and Non-Representative Cost Information

As with its effectiveness outcome, FMCSA identified improved efficiency in its Strategic Plan: Fiscal Years 2015–2018 as one of its strategic outcomes for CSA interventions. We have previously reported that agencies should measure program efficiency by considering the relationship between two elements: (1) inputs, such as costs or hours worked, and (2) desired results, such as a program’s effect on conditions or behaviors. In the past, FMCSA evaluated the efficiency of CSA interventions using both of these elements. Specifically, UMTRI’s August 2011 evaluation estimated the average cost of conducting individual intervention types and measured the effects on carrier safety performance of those CSA intervention types, as well as common intervention patterns, in four pilot test states during an 8-month period from October 2008 through May 2009. Since the UMTRI evaluation, FMCSA has continued to evaluate the effectiveness of CSA interventions. However, as we describe below, FMCSA has not taken similar steps to update its cost information—information FMCSA would need to understand the relationship between both efficiency elements.

The UMTRI evaluation developed average cost estimates for each intervention type by considering four cost variables: labor hours, government miles traveled, vouchers, and other expenses. The evaluation studied 920 interventions applied to 586 carriers, with the number of carriers receiving each intervention type ranging from 6 carriers for notices of violation to 249 carriers for onsite focused investigations. The evaluation concluded that cooperative safety plans and notices of violation had the lowest average estimated costs—ranging from $95 to $118 respectively. Onsite comprehensive investigations and onsite focused investigations had the highest estimated costs, averaging $1,038 and $677 respectively.

Since UMTRI conducted its August 2011 efficiency evaluation, FMCSA has continued to use the results to report and estimate the efficiency of interventions. For example, in its Budget Estimates: Fiscal Year 2017 report, FMCSA requested $2.5 million and 50 additional Program Analysts to complete the last phase of the CSA program, including nationwide implementation of offsite investigations. To support this request, FMCSA stated that offsite investigations were “extremely efficient” and specifically cited the evaluation’s cost estimate. Similarly, FMCSA headquarters officials told us they currently use the UMTRI evaluation to estimate efficiency and to understand the relative costs of individual intervention types.

However, cost estimates from UMTRI’s August 2011 evaluation are no longer current, because the time and resources needed to conduct interventions has changed and the evaluation’s estimates are not representative of all states.

The evaluation’s estimates are no longer current because the time and resources needed to conduct interventions has changed. For example, in April 2013, FMCSA implemented a substantial change to the way investigators conduct investigations, called EIT, which is intended to help identify the root cause of motor carriers’ safety problems. FMCSA headquarters officials told us that, although using EIT takes additional time, it results in improved motor carrier safety performance.

The evaluation’s estimates do not represent costs in all states.

Specifically, the evaluation stated that its estimates pertain only to the four states studied. Thus the evaluation’s estimates may not appropriately represent the average costs associated with applying interventions in the remaining 46 states, the District of Columbia, and Puerto Rico.

Federal standards for internal control state that agencies should use quality information to achieve the agency’s intended program outcomes. Quality information includes information that is current. FMCSA headquarters officials told us that they have not taken steps to update the cost estimates from the UMTRI evaluation to determine current resources used in all states to conduct each intervention type because they believed that FMCSA policy and guidance were sufficiently well designed to enable division managers to select the least resource-intensive intervention type necessary to correct a carrier’s safety problem.

In March 2015, FMCSA’s then-Acting Administrator testified that given FMCSA’s limited resources relative to the size of the regulated motor carrier population, it is imperative for FMCSA to apply its resources efficiently. However, without cost estimates for CSA interventions that are current and representative of all states, FMCSA lacks information it needs to understand the most efficient methods of conducting CSA interventions in all states. Because FMCSA lacks current cost information, it also cannot evaluate or understand the relationship between these costs and the effectiveness of CSA interventions.

FMCSA Took Some Steps Intended to Improve CSA Intervention Outcomes, but Lacks Measures to Monitor Progress

FMCSA has taken steps intended to improve the effectiveness and efficiency of CSA interventions—strategic outcomes—principally by establishing a working group to address these issues and implementing some of the group’s recommendations. However, FMCSA has not established performance measures to monitor progress toward achieving its intended efficiency outcome for interventions. Without establishing measures for both outcomes, FMCSA is and will be limited in its ability to monitor program performance and to balance these two priorities, as needed.

FMCSA Has Taken Some Steps Intended to Improve the Effectiveness and Efficiency of CSA Interventions

In April 2014, FMCSA formed a Continuous Improvement Working Group (CIWG) tasked with improving the effectiveness and efficiency of the CSA program, including CSA interventions. Comprised of FMCSA staff from divisions, service centers, and headquarters, as well as state partners, the CIWG’s objective was to assess the agency’s intervention and prioritization processes and to recommend improvements that increase program effectiveness and efficiency. To develop its recommendations, the working group reviewed available intervention data—such as investigation reports—and assessed current intervention practices by surveying and interviewing field staff, among other things. In February 2015, the CIWG made 20 recommendations intended to achieve its effectiveness and efficiency objectives. According to FMCSA officials, the agency had implemented 12 of these recommendations as of April 2016 and was working to implement the remaining 8 recommendations. Implemented recommendations include: Changing FMCSA’s high-risk definition and prioritization criteria: In March 2016, FMCSA adopted a change to its definition of high-risk carriers and the criteria it used to prioritize carriers for intervention. For example, as discussed above, for a carrier to be defined as high-risk under FMCSA’s new criteria, it has to exceed higher percentile thresholds than previously used in any of at least two of four specified BASICs. According to FMCSA, this decreases the total number of high-risk carriers but better identifies carriers at a high-risk for crashes and will allow investigators to investigate higher-risk carriers sooner, using current resources. However, FMCSA’s continued reliance on BASIC percentiles supported by insufficient safety data, which we identified in our 2014 report and discussed above, will limit its ability to effectively identify and prioritize the highest-risk carriers for intervention.

Changing criteria to receive warning letters: In January 2016, FMCSA expanded the criteria it used to determine which motor carriers receive warning letters in an effort to reach more carriers and, according to FMCSA, to prevent further non-compliance before a more intensive intervention type becomes necessary. Specifically, the CIWG recommended that FMCSA send warning letters to carriers with BASIC percentiles above the intervention threshold in more BASICs than previously allowed, while shortening the time period carriers have after receiving a warning letter to improve their safety performance before being prioritized for an investigation intervention. The CIWG projected that implementing the change would increase the number of warning letters that FMCSA issued by 30 percent; however, the CIWG noted that issuing a warning letter that is not effective may merely postpone the eventual necessity of a carrier receiving an investigation intervention and that issuing more frequent warning letters could dilute their effectiveness.

Establishing intervention quality review procedures: In March 2016, FMCSA issued two memorandums requiring field staff to use tools that FMCSA developed to evaluate and improve the quality of some intervention types. Specifically, FMCSA developed one tool to measure the extent to which investigators appropriately and accurately conducted onsite focused and comprehensive investigations and completed required documentation. Division managers are expected to use the tool to evaluate a selected sample of investigation reports quarterly, identify areas of needed improvement, and provide training to improve report consistency and quality. Similarly, FMCSA developed another tool that measures the extent to which investigators’ enforcement cases for notices of claim and notices of violation include sufficient documentation to meet evidentiary requirements. The memorandum requires division managers to ensure that each notice is reviewed to identify areas for improvement and ensure that enforcement cases are properly completed. According to FMCSA officials, FMCSA intends to use the results of these evaluations to identify training or policy clarifications needed to continuously improve the application and effectiveness of each intervention performed.

FMCSA Is Limited in Its Ability to Monitor Progress toward Achieving CSA Intervention Outcomes Because It Lacks Performance Measures

As previously discussed, FMCSA’s Strategic Plan: Fiscal Years 2015– 2018 identified improved effectiveness and efficiency as strategic outcomes of CSA interventions. FMCSA headquarters officials told us that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. For example, according to officials, while using EIT requires more time and decreases the number of investigations that FMCSA can conduct (i.e., efficiency), it also increases investigation quality (i.e., effectiveness). Thus, senior FMCSA officials stressed the importance of considering both effectiveness and efficiency in any set of measures used to monitor interventions, and stated that without treating these two outcomes as parts of a whole, FMCSA cannot achieve its goals for CSA interventions.

FMCSA has established some measures for its effectiveness outcome, and monitors these measures on an annual and ongoing basis. While we identified several limitations with the design and methodology FMCSA used in its effectiveness evaluation above, FMCSA has established a measure for intervention effectiveness—crash rates—and annually monitors the agency’s performance against that measure. Officials told us that FMCSA also monitors effectiveness using investigation outcome measures, such as violation rates and safety ratings, on an ongoing basis.

However, FMCSA has not established measures to monitor progress toward achieving its efficiency outcome. According to headquarters officials, FMCSA considers the efficiency outcome to include two dimensions: (1) the number of carriers FMCSA reaches through interventions and (2) the resources required for FMCSA to conduct interventions, including the time and travel required to complete investigations. While officials stated that FMCSA monitors some information related to efficiency, such as the number of investigations completed and investigative outcomes, officials acknowledged that FMCSA has not formally established measures for its efficiency outcome.

Leading practices for performance management state that agencies should express outcomes in a measurable form and establish a set of performance measures that help monitor progress toward achieving each outcome. Additionally, our work has shown that agencies should create a set of performance measures that addresses important dimensions of program performance and balances competing priorities to increase the usefulness of performance plans in guiding decisions. While our past work has identified challenges some federal agencies face developing and using outcome-based efficiency measures, it has also highlighted the importance of developing such measures.

Because FMCSA does not have a complete set of measures that reflects both its effectiveness and efficiency outcomes for CSA interventions, FMCSA managers lack benchmarks needed to regularly monitor progress toward achieving the outcomes. FMCSA also lacks information needed to balance these priorities and guide management decisions about FMCSA’s application of interventions. FMCSA’s limited resources and the increase in crashes involving motor carriers in recent years highlight the importance of ensuring that FMCSA regularly measures and monitors progress toward achieving both of these strategic outcomes.

Conclusions

Fatalities involving motor carriers have increased—rising from about 4,200 in 2011 to almost 4,500 in 2015—and interventions can play a critical role in reversing this troubling trend. FMCSA aims to reduce such fatalities by using a data-driven approach that identifies and intervenes with the highest-risk motor carriers early. To monitor its performance, FMCSA has identified improved effectiveness and efficiency as strategic outcomes for CSA interventions and has taken some steps to improve agency performance in these areas. For example, since March 2016, FMCSA has required field staff to use tools that FMCSA developed to evaluate and improve the quality of onsite investigations and two follow- on interventions.

However, we identified important limitations in the information that FMCSA used to evaluate the effectiveness and efficiency of interventions. For example, FMCSA’s effectiveness evaluations did not produce sufficiently complete, appropriate, and accurate information on individual intervention types or common intervention patterns, because of design and methodology limitations. Although FMCSA officials expressed concern about potential sample size limitations when evaluating the effectiveness of individual intervention types or common intervention patterns, the August 2011 UMTRI evaluation that FMCSA sponsored as part of its Operational Model Test demonstrated that such evaluations are feasible. Similarly, FMCSA uses cost estimates from UMTRI’s evaluation to understand the efficiency benefits of interventions. However, the evaluation’s cost estimates are no longer current because the time and resources needed to conduct interventions have changed and are not representative of costs nationwide. Without current cost estimates that are representative of nationwide costs, FMCSA lacks information it needs to understand the most efficient methods of conducting CSA interventions and cannot assess the relationship between these costs and intervention effectiveness.

Moreover, long-standing delays in developing SENTRI software have compromised the quality of intervention information, thereby limiting FMCSA’s ability to accurately and effectively monitor trends in its application of interventions over time and to evaluate intervention effectiveness. As FMCSA continues its efforts to address data limitations that affect the accuracy of intervention information, it is important that FMCSA not delay taking steps to improve how it currently evaluates the effectiveness and efficiency of CSA interventions by ensuring, for example, that its annual effectiveness evaluation addresses other limitations we have identified.

FMCSA has dedicated significant resources to transition from a costly one-size-fits-all approach to a range of more effective and efficient interventions. However, without improving the quality of information that FMCSA uses to evaluate its performance, the agency will continue to lack the information it needs to determine the extent to which it is achieving these fundamental programmatic improvements.

In addition, we found that although FMCSA has established some performance measures for its effectiveness outcomes, the agency has not established measures to monitor progress toward achieving its efficiency outcome. FMCSA needs information on all dimensions of its effectiveness and efficiency outcomes to balance these priorities and guide management decisions about its application of interventions.

Recommendations for Executive Action

To determine whether CSA interventions influence motor carrier safety performance, the Secretary of Transportation should direct the FMCSA Administrator to: Identify and implement, as appropriate, methods to evaluate the effectiveness of individual intervention types or common intervention patterns to obtain more complete, appropriate, and accurate information on the effectiveness of interventions in improving motor carrier safety performance. In identifying and implementing appropriate methods, FMCSA should incorporate accepted practices for designing program effectiveness evaluations, including practices that would enable FMCSA to more confidently attribute changes in carriers’ safety behavior to CSA interventions.

To understand the efficiency of CSA interventions the Secretary of Transportation should direct the FMCSA Administrator to:

Update FMCSA’s cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states.

To enable FMCSA management to monitor the agency’s progress in achieving its effectiveness and efficiency outcomes for CSA interventions and balance priorities, the Secretary of Transportation should direct the FMCSA Administrator to:

Establish and use performance measures to regularly monitor progress toward both FMCSA’s effectiveness outcome and its efficiency outcome.

Agency Comments

We provided a draft of this report to the DOT for review and comment. DOT provided written comments, which are reprinted in appendix IV. In its written comments, DOT concurred with our recommendations. DOT also described actions that FMCSA has taken to improve the CSA program and noted that CSA interventions have been shown to effectively improve motor carriers’ safety behavior. As stated in this report, the evaluations that FMCSA uses to assess intervention effectiveness did not produce sufficiently complete, appropriate, and accurate information on individual intervention types because of design and methodological limitations that limited FMCSA’s ability to accurately attribute changes in carriers’ safety behavior solely to interventions. We believe that identifying and implementing appropriate methods to address these limitations will help to provide FMCSA with information it needs to evaluate its performance in achieving its effectiveness outcome for CSA interventions.

In addition, DOT provided technical comments on the draft report, which we incorporated as appropriate.

We are sending copies of this report to relevant congressional committees, the Secretary of Transportation, and the Administrator of FMCSA. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of our report were to examine: (1) the extent to which the Federal Motor Carrier Safety Administration (FMCSA) has implemented Compliance, Safety, Accountability (CSA) interventions and how it has applied them; (2) the extent to which FMCSA has evaluated the effectiveness and efficiency of CSA interventions; and (3) any steps that FMCSA has taken to improve and monitor progress toward achieving its intended outcomes for CSA interventions.

To determine the extent to which FMCSA has implemented CSA program interventions and how it applied them, we analyzed FMCSA intervention data from fiscal year 2010 through fiscal year 2015, the most recent fiscal year for which intervention information was available. Specifically, we analyzed data from two FMCSA data systems: (1) the Motor Carrier Management Information System (MCMIS) and (2) the Enforcement Management Information System (EMIS). If we determined that MCMIS and EMIS data were sufficiently reliable, we intended to then analyze whether there were any notable increases, decreases, or other trends in FMCSA’s application of interventions—across states, regions, and motor carrier types (e.g., fleet size). We also intended to determine common intervention patterns when carriers receive multiple interventions (e.g., warning letter, then off-site investigation, then notice of violation).

To determine the reliability of FMCSA data we requested a complete set of all MCMIS and EMIS data from fiscal year 2010 through fiscal year 2015. To develop our request, we conducted interviews with cognizant FMCSA officials as well as officials from the Volpe National Transportation Systems Center, which provides technical support to FMCSA’s data analysis. We would typically review documentation to prepare a data analysis plan; however, FMCSA could not provide up-to- date or complete data dictionaries or other reference documents for these data systems. Thus, we requested and FMCSA provided sample data tables and variables that we could use to identify the occurrence of each intervention type by analyzing MCMIS and EMIS data.

Using the information that FMCSA provided, we performed electronic data testing to count how frequently FMCSA conducted each intervention type in fiscal year 2010 through fiscal year 2015. We then compared the results of our analysis against FMCSA-published sources to determine if the results included obvious errors or outliers. While we replicated FMCSA’s counts for warning letters and unsatisfactory/unfit out-of-service orders in all fiscal years, our comparison revealed differences in at least one fiscal year for all remaining intervention types. We subsequently met with FMCSA data analysis officials on several occasions to identify the cause of the differences that we identified. FMCSA officials told us that when they modified the algorithms used to count interventions, the agency applied the most recent algorithm to all previous data—including historical data—thereby changing the way that FMCSA counted interventions over time. As a result, FMCSA officials told us that we could not validate the results of our analysis against agency totals.

After evaluating the reliability of these data for our analytical and reporting purposes, we concluded that the data were of undetermined reliability, because data limitations prevented an adequate and comprehensive assessment. This precluded our analysis of trends in FMCSA’s application of interventions across states, regions, and motor carrier types. As a substitute, we requested that FMCSA provide estimates for how frequently it applied each intervention type from fiscal year 2010 through fiscal year 2015 to identify general trends. After reviewing FMCSA documentation related to the estimates and interviewing responsible FMCSA officials, we determined that FMCSA’s estimates were sufficiently reliable for this purpose.

We reviewed relevant regulations and FMCSA guidance and policy documents to identify how FMCSA should implement and apply interventions. For example, we reviewed the electronic Field Operations Training Manual (eFOTM), which is the principal guidance document for assigning intervention types.

In addition, we interviewed FMCSA division officials in eight states that we selected based upon their participation in FMCSA’s Operational Model Test of the CSA program, geographic location, and program size, among other factors. Selected states included:

Georgia

Illinois

Kansas

Maryland

Massachusetts

Montana

Oklahoma

Texas We selected these states to get a range of perspectives on FMCSA’s application of interventions. For example, four of the states participated in FMCSA’s Operational Model Test and thus had experience implementing all eight CSA intervention types. Similarly, we selected two states from each service center. Although the information obtained from our interviews with officials from the selected states is not generalizable to all states or FMCSA divisions, it provided illustrative examples of how FMCSA is applying interventions as well as the perspectives of officials knowledgeable about the program. In addition, we interviewed FMCSA officials from each service center—including FMCSA’s Eastern, Midwestern, Southern, and Western Service Centers—as well as headquarters officials from FMCSA’s Office of Enforcement, Office of Field Operations, and Office of Research and Information Technology. We also interviewed industry stakeholders, such as the Commercial Vehicle Safety Alliance, American Trucking Associations, Trucking Alliance, and the Owner-Operator Independent Drivers Association to gain their perspectives on FMCSA’s intervention and enforcement activities.

To determine the extent to which FMCSA has evaluated the effectiveness and efficiency of CSA interventions, we intended to conduct our own effectiveness evaluation to determine how interventions affect motor carrier safety and illustrate the strengths and limitations of particular evaluation designs. However, because MCMIS and EMIS data were of undetermined reliability, we instead reviewed the four evaluations the agency has conducted. Specifically, we reviewed:

The University of Michigan Transportation Research Institute, Evaluation of the CSA 2010 Operational Model Test (August 2011);

FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier Intervention Effectiveness Model, Version 1.0: Summary Report for Fiscal Years 2009, 2010, 2011 (January 2015);

FMCSA, Analysis Brief: Effectiveness of Onsite Focused Investigations (March 2016); and

FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis (March 2016)

We conducted a more detailed assessment of the second report on FMCSA’s annual effectiveness evaluation model because, according to FMCSA headquarters officials, it is the primary method FMCSA uses to evaluate intervention effectiveness and FMCSA intends to use it on a recurring basis. In addition, we reviewed FMCSA policy documents— such as FMCSA’s Strategic Plan: Fiscal Years 2015–2018 and eFOTM guidance—to determine how FMCSA used the information produced by each of its four evaluations. We also interviewed FMCSA headquarters officials responsible for developing policy and conducting data analysis as well as officials from the Volpe National Transportation Systems Center responsible for designing and conducting some FMCSA evaluations.

To assess FMCSA’s effectiveness evaluations, we consulted GAO’s guidance on designing program evaluations, which describes accepted practices for evaluating program effectiveness based on GAO studies, policy documents, and program evaluation literature. We also consulted federal standards for internal control for using quality information, and drew on internal methodological expertise to assess the extent to which the designs, implementation, and results of FMCSA’s evaluations met quality information standards.

To determine any steps that FMCSA has taken to improve and monitor progress toward achieving its intended outcomes for interventions, we reviewed relevant FMCSA strategic planning and policy documents that established such outcomes, principally FMCSA’s Strategic Plan: Fiscal Years 2015–2018. We reviewed reports from external entities that included recommendations intended to support improvements to FMCSA’s compliance and enforcement programs, such as the Independent Review Team’s July 2014 report and a March 2014 report from the Department of Transportation’s Office of Inspector General.

Similarly, we reviewed FMCSA’s Continuous Improvement Working Group’s February 2015 report that included recommendations intended to achieve FMCSA’s effectiveness and efficiency outcomes.

We also interviewed responsible division, service center, and headquarters officials to identify any steps FMCSA has taken to monitor or improve the effectiveness or efficiency of interventions as well as to determine their perspectives on the effects of these steps. For example, we interviewed responsible officials at headquarters who develop policy and oversee adherence to federal regulations by interstate motor carriers across the country to understand how they monitor or improve interventions. We also interviewed service center and division officials to understand the field’s involvement in FMCSA’s improvement activities. We compared the results of our documentary review and interviews against leading practices for performance management as well as key attributes of successful performance measures identified in our prior body of work. Although GPRAMA’s requirements apply at the departmental level (e.g., the Department of Transportation), we have previously stated they can serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects.

We conducted this performance audit from July 2015 to October 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on the audit objectives.

Appendix II: Obligations to Develop Safety Enforcement Tracking and Investigation System Software

Headquarters officials told us that the Federal Motor Carrier Safety Administration (FMCSA) obligated funds to develop Safety Enforcement Tracking and Investigation System (SENTRI) software each year from fiscal year 2009 through fiscal year 2013; however, they could not determine the exact amount of funds because FMCSA did not track information technology investments at the project level during those years. In addition, the agency obligated about $12 million from fiscal year 2014 through fiscal year 2016 in contractor costs to develop the Compliance, Safety, Accountability (CSA) component of SENTRI (see table 5). FMCSA officials told us that FMCSA made some progress in developing the CSA component of SENTRI as a result of these investments. For example, FMCSA coordinated with field staff to identify system requirements.

Appendix III: Analysis of the Federal Motor Carrier Safety Administration’s Carrier Intervention Effectiveness Model

The Federal Motor Carrier Safety Administration (FMCSA), in conjunction with the John A. Volpe National Transportation Systems Center (Volpe) has modified an existing effectiveness model to develop a statistical model, the Carrier Intervention Effectiveness Model (CIEM), which annually measures the effectiveness of interventions. In January 2015, FMCSA published its first report using the CIEM, which evaluated the effectiveness of compliance reviews, and interventions FMCSA conducted in fiscal years 2009, 2010, and 2011. We analyzed the CIEM and the January 2015 evaluation report, using accepted practices for designing program evaluations and internal staff expertise. Below, we identify methodological strengths and limitations of these efforts, in addition to potential methods FMCSA could use to improve the capabilities of its model to estimate program impacts. Specifically, we have identified strengths and limitations in four key aspects of FMCSA’s effectiveness model: (1) general design, (2) comparison group, (3) observation periods, and (4) statistical analysis and inference.

General Design

According to FMCSA documentation, the CIEM is a statistical impact evaluation model that uses historical data to compare the safety improvement of carriers receiving FMCSA interventions (the treatment group) to carriers that do not (the comparison group). The January 2015 evaluation report assessed two intervention types that existed prior to the Compliance, Safety, Accountability (CSA) program, including compliance review investigations and Performance and Registration Information Systems Management letters, and four new CSA intervention types, including warning letters, offsite investigations, onsite focused investigations, and onsite comprehensive investigations.

To estimate the impact of these interventions, the CIEM measures the difference between crash rates among carriers in the treatment group before and after receiving interventions, and then subtracts the difference in crash rates among carriers in the comparison group. The comparison group accounts for confounding factors (other than FMCSA interventions) that may affect safety performance during the post-intervention period, such as broad changes in weather or economic conditions.

The model is designed to estimate the impact of interventions carried out in a single fiscal year and measures crash rates over a 12-month period following the first intervention a carrier receives in the fiscal year. The model estimates the combined impact of all interventions performed during 12-month periods, not the unique impact of each individual intervention type.

We identified the following limitations:

Lack of process evaluation: The CIEM is designed to evaluate impact, but does not include a process study. According to accepted practices for designing program evaluations, a program logic model or process evaluation that identifies the most important external influences on desired program outcomes is valuable in planning an impact evaluation that convincingly rules out most plausible alternative explanations for the observed results. Process evaluations clarify the program as implemented and specify which of its activities may be responsible for the observed outcomes. The CSA program’s logic model might include elements of the CSA program, such as FMCSA and state safety inspectors and information systems, roadside inspections, safety interventions, and crashes, injuries, deaths, and monetary losses prevented.

Without an initial process evaluation, the impact evaluation cannot precisely identify what aspects of the program affect safety outcomes, or whether the estimated impacts reflect the program as designed. Without studying the program’s implementation and comparing its theoretical logic model to actual practices, it is uncertain whether impact estimates represent the effectiveness of the program’s activities as designed or the activities that program staff happened to have used in practice. The CIEM’s ability to accurately evaluate the impact of the CSA program could be improved by taking into account the program’s strategy and goals, and studying how the program is implemented.

Exclusion of intervention types: The January 2015 evaluation report did not include all intervention types. According to the report, the evaluation did not assess cooperative safety plans or direct notices of violation and direct notices of claim, because data on these intervention types had inconsistent completeness and accuracy. FMCSA officials told us that the evaluation included carriers that received follow-on notices of violation or claim, but it did not separately estimate their effects. When notices of violation or claim follow an investigation, the model implicitly estimates their effects in the post-intervention crash rate. However, the evaluation could have excluded some carriers that received a notice as their first intervention in the modeled year but not in the same fiscal year as the investigation.

Aggregation of intervention types: The CIEM implicitly estimates how combinations of interventions affect safety, not the effects of each individual intervention type. The model identifies only the first intervention that a carrier receives in the modeled year, and estimates the intervention’s effect on crash rates from that time of first contact.

According to FMCSA officials, the model is designed to evaluate the effect of general FMCSA contact with carriers through interventions. Officials said that they tested some alternatives to using the first intervention in the fiscal year, including using a carrier’s most severe or last intervention in the fiscal year. Agency staff told us that they ultimately preferred to use the first intervention, because it represented the beginning of FMCSA’s influence on a carrier within the target time period.

FMCSA did not seek to estimate the effectiveness of individual intervention types, because agency staff had concerns about the small amount of data available on specific intervention types and common intervention patterns when carriers receive multiple interventions over time. FMCSA officials stated that Volpe conducted preliminary analysis of the data used for the CIEM to determine potential sample sizes, but did not provide documentation on the sample sizes for each intervention type or common patterns of interventions. The challenges arising from small sample sizes could potentially be addressed by pooling together data from multiple years, rather than relying on intervention data from one fiscal year.

Recommended practices of program and policy evaluation recommend that program managers attempt to separately evaluate multiple types of program activities that seek to achieve a common outcome. Such “comparative effectiveness (or efficiency)” evaluations give managers information on how various activities perform compared to each other. Variation in outcomes across settings or populations can be the result of variation in program operations, such as varying types or levels of enforcement. Further, variation in outcomes associated with features under program control, such as different agency activities, may identify opportunities for managers to take action to improve performance.

To obtain this kind of information on comparative effectiveness, the treatment effects of interventions could be disaggregated into more nuanced categories than having at least one intervention. This approach could directly evaluate how each of several intervention types, or common intervention patterns, affect safety outcomes. Detailed performance information would better align with the design of the CSA program and give staff flexibility to choose from a range of intervention types that can address each carrier’s unique safety problems. With evidence of comparative effectiveness and efficiency, FMCSA would have better information on whether specific CSA interventions or combinations of interventions are more effective than others in certain circumstances.

The challenges from small sample sizes on particular interventions could be addressed in several ways. Pooling together data from multiple years, rather than relying on intervention data from one fiscal year, might produce a sufficient sample for evaluating less commonly used interventions. A multi-year design might become viable as the CSA program continues to produce data over several years, though pooling data might increase the potential for unmeasured factors to influence safety outcomes. A process evaluation, as discussed above, could clarify how FMCSA field staff and state partners have implemented the program and could identify intervention types, or specific combinations of interventions, with sufficient data for analysis.

Regression to the mean: The CIEM design does not fully account for the possibility that variation over time in the treatment group’s safety outcomes reflects regression to the mean. Regression to the mean is a statistical phenomenon in which, following an extreme measurement assumed to be due to random sources of variation, such as sampling error, subsequent measurements are likely to be closer to the average, or mean.

Under the CSA program, FMCSA prioritizes carriers to receive interventions based on whether the carriers’ percentiles exceed pre- determined thresholds in any of seven behavioral analysis and safety improvement categories (BASIC). FMCSA’s decision to intervene with a carrier largely depends on BASIC percentiles, and officials use these percentiles, in addition to other criteria set out in guidance, to determine which type of intervention to apply. This is especially true for carriers that exceed the thresholds intended to identify the highest risk carriers, because FMCSA policy requires that those carriers receive onsite investigations.

However, some carriers, especially those with few inspections or vehicles, may go above the intervention threshold in one measurement period due to anomalous events, but return below the threshold in the next measurement period due to factors unrelated to intervention. For example, a carrier may generally violate vehicle maintenance regulations at the industry average rate over the long- term. The carrier’s estimated violation rate in FMCSA data—and its related BASIC percentile—may vary around the long-term average in any particular period. The degree of variation could reflect the actual inconsistency of the carrier’s maintenance practices over time or sampling error in the estimation of its violation rate, related to its frequency of inspection (exposure to violating regulations). A large deviation in one period from the long-term average could exceed BASIC percentile thresholds and trigger additional FMCSA oversight, but the deviation may not reflect a real change in the carrier’s long- term maintenance behavior. In a subsequent period, the carrier’s BASIC percentile has a higher probability of returning to the long-term average than continuing at the extreme from the previous period, assuming the prior deviation was caused by random sources of variation, such as sampling error. The same process may apply to crash rates.

According to recommended practices for evaluating the impact of a program, the evaluation must be carefully designed to rule out plausible alternative explanations for the results. The CIEM’s quasi- experiment design implicitly controls for differences across carriers that do not vary substantially over short time periods, which could include carrier management practices, leadership, and operating routes and procedures. The design controls for industry-wide changes over time that are constant across carriers, changes that could include weather and economic conditions. Lastly, the design explicitly controls for carrier size by stratifying the analysis by size groups. Although the size control may account for differences within carriers over time among the treatment carriers, the model includes few other controls that might specifically address this potential threat to valid causal inference. By not fully accounting for regression to the mean, the CIEM could be attributing changes in outcomes to CSA interventions, when those changes would have occurred on their own without intervention.

Accordingly, there is some evidence that some carriers’ safety behavior improves without intervention. For example, the Independent Review Team found that, of the carriers FMCSA prioritized for intervention in 2013, nearly 33 percent had a BASIC percentile above threshold when assigned for intervention that was no longer above threshold at the time of the review, suggesting that carriers’ safety performance may improve naturally without intervention. As we have previously reported, carriers with fewer vehicles experience wide variance in crash and violation rates, which can make them especially prone to regression to the mean.

An alternative design might compare carriers just above and just below the intervention-triggering threshold (at a given point in time). This “regression discontinuity” design would lend itself better to interventions triggered automatically when carriers exceed some threshold and would better reflect the nature of the CSA program, as recommended in accepted practices for evaluation design.

Another alternative approach that could address the regression to the mean issue would be to match carriers based on variation over time in the safety outcomes prior to exceeding a BASIC percentile threshold. This design would fall into a general class of methods that include pre-treatment outcomes as an additional covariate. This would enhance the comparison group’s control for any deviations in the outcome over time prior to treatment and thereby ensure that treatment carriers would be matched to comparison carriers with similar outcome dynamics.

Comparison Group

The CIEM constructs comparison groups using carriers that did not receive any of the model’s interventions during the modeled, prior, or subsequent year. The model assigns carriers to one of several comparison groups, using the same measure of size—the number of vehicles—used to assign carriers to treatment groups. These separate comparison groups are intended to eliminate differences associated with carrier size from the model’s calculation of adjusted crash rates.

We identified the following limitation:

Limited control: Accepted practices for program evaluation would classify the CIEM as a “quasi-experimental” design for estimating impact. Quasi-experimental designs compare outcomes in a treatment group to outcomes in a comparison group formed using non-random assignment. Due to the lack of randomized assignment, the treatment and comparison groups may differ on other factors that affect the outcome. To compensate for this potential bias, evaluators generally ensure that such confounding variables are held constant in the construction of the comparison group or use other methods of adjustment.

The CIEM explicitly holds constant one factor in the construction of its comparison group: carrier size. The model implicitly controls for all factors that vary between the treatment and control groups but remain constant over time, such as state or region of operation. In addition, the model implicitly controls for all factors that vary over time and affect the treatment and comparison groups equivalently, such as industry-wide effects due to weather or economic conditions.

The CIEM’s “difference-in-difference” design provides these forms of implicit control, and thus is a key strength of the model. However, FMCSA might try to construct a more robust comparison group that explicitly controls for more than just carrier size. Since the design controls for variables that are fixed across carriers and vary in the same ways within groups over time, FMCSA might construct a comparison group that controls for change across multiple variables. If reliable data were available, then potential variables could include: multiple measures of size; state of registration, inspections, or violations; driver characteristics; pre-treatment safety outcomes; pre- treatment BASIC percentiles; and pre-treatment inspections and regulatory violations. Controlling for pre-treatment outcomes and BASIC percentiles would be especially desirable and would address the potential limitation of regression to the mean because the fluctuations due to sampling error would be controlled by design.

Data availability and reliability may limit FMCSA’s ability to include additional control variables to construct comparison groups. For example, the 2015 evaluation report notes that data on vehicle miles traveled—a measure of carrier exposure to crashes—were less reliable than vehicle count data because, according to officials, they were sometimes incomplete or inconsistent across carriers. Similarly, officials told us that FMCSA considered using carrier operation type to construct comparison groups, but ultimately did not because some carriers reported multiple operation types or changed their operation types over time. Given the central importance of the CSA program to FMCSA’s enforcement efforts, the agency would benefit from improving or expanding data collection to support a more robust model.

Observation Periods

The CIEM defines different observation periods for the treatment and control groups. For the treatment group, the CIEM uses the date of the first intervention in the modeled fiscal year as the demarcation between a 12-month pre-intervention period and a 12-month post-intervention period. Pre- and post-intervention crash rates are measured for these 12- month periods. For the comparison group, in contrast, the CIEM defines 18-month periods preceding and following the midpoint of the modeled fiscal year to measure pre- and post-intervention crash rates. (This is because comparison carriers do not have an intervention date to define the pre- and post-intervention periods and measure crash rates.) The evaluation report noted that these longer 18-month periods ensure that the comparison group’s crash rates cover the entire treatment group time frame. To adjust for the 50 percent longer observation period for carriers in the comparison group, the evaluation divided crash rates for those carriers by 1.5 to yield annual crash rates.

We identified the following limitation: Inconsistent time periods: According to accepted practices for program impact evaluation, a design must confidently rule out non- program influences that could cause changes in outcomes to occur. The CIEM does not completely account for factors that might have affected crash rates for both treatment and comparison groups, because the lack of overlap between the measurement time periods does not hold constant factors unique to the season or period of measurement. For example, as FMCSA officials noted to us, crash rates are known to depend on seasonal and periodic changes in weather, and the model’s lack of overlap in measurement time periods meant that treatment and comparison groups were subject to different seasonal and periodic conditions. Other potential confounding variables include seasonal or periodic variation in economic demand and state enforcement resources.

The design might use measurement periods that vary as a function of each observed intervention’s timing and characteristics. For example, the design might construct a custom comparison group for each member of the treatment group, selecting multiple comparison carriers using size and other potential confounding variables. A matching design in which each treatment carrier is compared to one or more control carriers would allow identical observation periods while still producing an impact estimate for interventions applied during a single fiscal year.

Statistical Analysis and Inference

The CIEM’s use of statistical inference is appropriate to quantify the uncertainty of its impact estimates. Carrier behavior and safety outcomes are consistent with a partially random data generation process. In this context, statistical inference estimates the sampling variability of the data over multiple hypothetical realizations of the same process. Applying inferential statistical methods appropriately reflects the potential for the observational regulatory, intervention, and safety data to vary partially at random.

A critical estimate in the CIEM is the net crash rate change for the treatment group. The model defines this quantity as the difference between the treatment group’s pre- and post-intervention crash rates, after subtracting the crash rate change in the comparison group. The model then tests whether the net change differs from zero at the 0.05 statistical significance level. The model excludes insignificant findings from its subsequent estimates of total safety benefits calculated.

The model estimates safety benefits by transforming the estimated net change in crash rates due to interventions into an estimate of total crashes avoided, using the treatment group’s pre-intervention crash rate per vehicle and post-intervention vehicle counts. The model uses historical crash severity data to further estimate injuries prevented and lives saved associated with each prevented crash.

The model extrapolates these safety benefits—crashes avoided, injuries prevented, and lives saved—to carriers that received interventions, but were excluded from the treatment group due to missing or outlier crash or vehicle count data, and, according to officials, intrastate carriers that were excluded from the treatment group in the January 2015 version of the model. The January 2015 evaluation notes that FMCSA assumed that these carriers will exhibit the same response to intervention as the carriers included in the model. Accordingly, the model adds the estimated safety benefits for carriers included in the model to those for carriers with outliers and missing data. The sum determines the aggregate estimated safety benefits. According to the 2015 evaluation report, FMCSA extrapolated safety benefits to the following number of carriers that received an intervention but were not included in the model: 9,567 carriers in fiscal year 2009; 9,929 carriers in fiscal year 2010, and; 14,816 carriers in fiscal year 2011.

We identified the following limitations:

Multiple hypothesis tests: The model uses multiple statistical hypothesis tests to calculate total safety benefits. The CIEM estimates impact on crash rates within four strata of treatment and comparison groups defined by carrier size. If the net change in crash rates within each stratum is statistically distinguishable from zero at the 0.05 confidence level, the model uses the results to estimate total safety benefits by summing the estimated benefits across groups. Statistically insignificant results in any stratum provide zero safety benefits by assumption. In this sense, the model’s estimate of total safety benefit reflects the results of 4 separate hypothesis tests, each at the 0.05 confidence level. = 1 – (1 – α)K. When K = 4 and α = 0.05, α = 0.18. See Hayes, 450. methods of hypothesis testing typically recommend adjusting the confidence level of each individual test, such as by applying Bonferroni adjustments, so that the group-wise error probability matches the analyst’s intended risk level for all planned tests. These adjustments typically produce lower alpha values for each individual test.

Multiple hypothesis testing methods would be appropriate for the CIEM, given that it ultimately seeks to estimate total safety benefits as a function of multiple hypothesis tests. Without making these adjustments, the CIEM’s estimates of total safety benefits may have more risk of error than FMCSA intends to accept because the model does not conduct a joint test. That is, the probability that at least one group’s safety benefits equals zero may exceed the 0.05 confidence level that FMCSA accepts in each individual test. An alternative approach might calculate the confidence interval of the summed safety benefit estimate and test whether it equals zero, consistent with the discussion below.

Confidence intervals of estimated impacts: The CIEM uses inferential statistical methods to test the hypothesis that the impact for each pair of treatment and control groups equals zero. However, the CIEM does not conduct statistical inference, such as estimating confidence intervals, when analyzing the net change in crash rates and associated total safety benefits. This approach is inconsistent, given that the model’s hypothesis test for a non-zero net change in crash rates implies that the quantity is a random variable with a sampling distribution and confidence interval. The model’s authors agree with this implication, suggesting that the test is equivalent to a “95 percent confidence interval that does not include zero.” Total safety benefits must also have a confidence interval, given that it is a function of the change in net crash rates. Nevertheless, the CIEM reports only point estimates of safety benefits, without conveying the statistical uncertainty that the model’s assumptions imply.

Accordingly, the CIEM might estimate and report confidence intervals for its estimates of crash rate impact and safety benefits, in order to make the statistical inference consistent and quantify the uncertainty of its estimates. The current hypothesis testing approach may produce a point estimate for safety benefits that appears more precise than the underlying confidence interval would support.

Appendix IV: Comments from the Department of Transportation

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, H. Brandon Haller (Assistant Director), William Colwell (Analyst in Charge), Katherine Blair, Melissa Bodeau, Russell Burnett, David Hooper, Benjamin Licht, Grant Mallie, Ifunanya Nwokedi, Malika Rice, Sandra Sokol, Niti Tandon, and Jeff Tessin made key contributions to this report.