Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions
Published: 2016-10-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As part of its mission to reduce crashes and fatalities involving large commercial trucks and buses, FMCSA seeks to use a data-driven approach to identify the highest-risk motor carriers and address safety problems by applying a range of eight CSA program enforcement tools, called interventions, ranging from warning letters to placing carriers out of service.
A provision in a Senate report requires GAO to periodically assess FMCSA's implementation of the CSA program. This report examines the extent to which FMCSA has (1) implemented CSA interventions, (2) evaluated the effectiveness and efficiency of CSA interventions, and (3) monitored progress toward achieving outcomes.
GAO reviewed FMCSA data and documentation on all eight CSA intervention types from fiscal years 2010–2015, including FMCSA's strategic planning documents, guidance, and program evaluations. GAO interviewed industry stakeholders and FMCSA officials in headquarters, in each of FMCSA's service centers, and in eight states selected for their participation in FMCSA's CSA pilot test, location, and program size, among other factors.
section_title: What GAO Found
paragraphs: In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing software needed to support the two interventions, offsite investigations and the use of cooperative safety plans. The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017.
FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA's approaches that impact the usefulness of the evaluations:
Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA's ability to accurately draw conclusions about intervention effectiveness across all intervention types.
Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions.
FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions, but lacks measures to monitor progress. In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.
section_title: What GAO Recommends
paragraphs: GAO recommends that FMCSA evaluate the effectiveness of individual intervention types, update cost estimates so that they are current and representative of all states, and establish complete performance measures.
The Department of Transportation concurred with all of GAO's recommendations.

section_title: Letter
paragraphs: Commercial trucks and buses are vital transportation links connecting  goods and people with American markets and communities. However, the  Federal Motor Carrier Safety Administration (FMCSA) reported that  crashes involving large commercial trucks and buses increased from  about 131,000 in 2011 to nearly 157,000 in 2015. These crashes harm  the well-being of the people involved and the companies that own  commercial trucks and buses (motor carriers). Fatalities resulting from  these crashes have also increased, rising from about 4,200 in 2011 to  almost 4,500 in 2015. Further, FMCSA estimated that crashes involving  large trucks and buses cost over $110 billion in 2014.
FMCSA’s Compliance, Safety, Accountability (CSA) program is intended  to reduce the number of motor carrier crashes by using a data-driven  approach to identify and intervene with the highest-risk motor carriers.  When motor carrier safety problems are identified, FMCSA applies a  range of intervention types—such as sending warning letters, conducting  investigations, or issuing civil penalties—that attempt to remedy those  problems early and before crashes occur.
A provision in a Senate Appropriations Committee report required us to  periodically assess FMCSA’s implementation of the CSA program. This  report examines: (1) the extent to which FMCSA has implemented CSA interventions and how it has applied them; (2) the extent to which FMCSA  has evaluated the effectiveness and efficiency of CSA interventions; and  (3) any steps that FMCSA has taken to improve and monitor progress  toward achieving its intended outcomes for CSA interventions.
To determine the extent to which FMCSA has implemented CSA  interventions and how it has applied them, we analyzed FMCSA  intervention data from fiscal year 2010 through fiscal year 2015. We  intended to analyze whether there were any notable increases,  decreases, or other trends in FMCSA’s application of interventions— across states, regions, and motor carrier types. However, data limitations  prevented an adequate and comprehensive assessment of the reliability  of FMCSA’s intervention data. For example, officials told us FMCSA  changed the way it counted interventions over time, and as a result, we  could not validate the results of our analysis against agency totals—a  step to testing data reliability. As a result, we concluded that FMCSA’s  intervention data were of undetermined reliability, a factor that precluded  our analysis of trends in FMCSA’s application of interventions across  states, regions, and motor carrier types. As a substitute, we requested  that FMCSA provide estimates for how frequently it applied each  intervention type from fiscal year 2010 through fiscal year 2015 to identify  general trends. After reviewing FMCSA documentation related to the  estimates and interviewing responsible FMCSA officials, we concluded  that FMCSA’s estimates were sufficiently reliable for this purpose. We  also reviewed relevant regulations and FMCSA guidance and policy  documents to identify how FMCSA should implement and apply  interventions. We interviewed headquarters officials from FMCSA’s Office  of Enforcement, Office of Field Operations, and its Office of Research and  Information Technology. FMCSA has 52 divisions that correspond with  each state, the District of Columbia, and Puerto Rico. One of the four  service centers oversees each division. We interviewed FMCSA division  officials in eight states that we selected based upon their participation in  FMCSA’s Operational Model Test of the CSA program, geographic  location, and program size, among other factors, and interviewed FMCSA  officials from each of the four service centers. Although the information  obtained from our interviews with officials from the selected states is not  generalizable to all states or FMCSA divisions, it provided illustrative  examples of how FMCSA is applying interventions as well as the  perspectives of officials knowledgeable about the program.
To determine the extent to which FMCSA has evaluated the effectiveness  and efficiency of CSA interventions, we reviewed the four evaluations the  agency has conducted. These included FMCSA’s January 2015  evaluation on intervention effectiveness; the University of Michigan  Transportation Research Institute’s (UMTRI) August 2011 evaluation of  the Operational Model Test, sponsored by FMCSA; and two effectiveness  evaluations on individual intervention types that FMCSA conducted in  March 2016. To assess these evaluations, we consulted accepted  practices for evaluation design and drew upon internal methodological  expertise. We also interviewed FMCSA headquarters officials  responsible for developing policy and conducting data analysis, as well as  Volpe National Transportation Systems Center officials responsible for  designing and conducting some FMCSA evaluations.
To determine any steps that FMCSA has taken to improve and monitor  progress toward achieving its intended outcomes for interventions, we  reviewed relevant FMCSA strategic planning and policy documents, such  as FMCSA’s Strategic Plan: Fiscal Years 2015–2018. We interviewed  responsible FMCSA division, service center, and headquarters officials to  identify any monitoring or improvement steps that FMCSA has taken as  well as to determine their perspectives on the effect of these steps. We  compared the results of our documentary review and interviews against  leading practices for performance management identified in our prior  body of work.
We conducted this performance audit from July 2015 to October 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on the audit objectives.
subsections: section_title: Background
paragraphs: FMCSA was established within the Department of Transportation (DOT)  in January 2000, and is tasked with promoting safe commercial motor  vehicle operations and reducing large truck and bus crashes, injuries, and  fatalities. It seeks to achieve this reduction through regulation,  enforcement, and partnerships with stakeholders, among other activities,  and with full accountability to the public through transparency, results- oriented performance measuring, and managing for results. Since fiscal  year 2010, FMCSA’s total budget authority to conduct these activities has  remained relatively stable, increasing about 6.5 percent from fiscal year  2010 through fiscal year 2016 (see table 1). Funding for implementing  and applying interventions is included in both the Motor Carrier Safety  Operations and Programs and Safety Grants budget authorities.
The vast majority of FMCSA’s staff are located in field offices, including  divisions and service centers. Field staff are primarily responsible for  implementing FMCSA’s compliance and enforcement activities, including  investigations. Federal Safety Investigators represent the majority of  FMCSA’s compliance and enforcement program staff (see table 2). In  addition, FMCSA partners with state agencies to perform some  intervention activities, such as conducting carrier investigations; however,  FMCSA is responsible for ensuring that commercial motor carriers under  its authority comply with federal safety regulations. FMCSA’s Office of  Enforcement is the primary office responsible for developing policy for  FMCSA’s compliance and enforcement program, and overseeing the  implementation of intervention activities.
subsections: section_title: The CSA Program
paragraphs: The CSA program is intended to improve the effectiveness of FMCSA’s  compliance and enforcement programs, while more efficiently using its  resources to reach carriers that pose the highest safety risk. The CSA  program has three key components: (1) the Safety Measurement System  (SMS), meant to identify high-risk carriers by using data from roadside  inspections and crashes; (2) interventions, which are intended to help  carriers address safety problems; and (3) the Safety Fitness  Determination rule. In contrast to the one investigation intervention, the  compliance review, available under FMCSA’s previous approach, FMCSA  expects the CSA program to more effectively change unsafe behaviors by  reaching and intervening with more potentially  unsafe carriers earlier  using a range of intervention types to enforce compliance with safety  regulations. In 2014, we reported on the effectiveness of the SMS  component of the CSA program.
FMCSA conducted an Operational Model Test of the program from  February 2008 through June 2010 in nine pilot states. In December  2010, FMCSA began implementing the CSA program in three phases: (1)  implementation of SMS and some intervention types nationwide, (2) introduction of new investigative techniques and the Safety Management  Cycle, and (3) nationwide rollout of all intervention types and FMCSA’s  new investigative software, the Safety Enforcement Tracking and  Investigation System (SENTRI).
After a series of four commercial motor vehicle crashes—two involving  buses and two involving trucks—that together resulted in 25 deaths and  injuries to 83 people, the National Transportation Safety Board  investigated and, in November 2013, made recommendations to improve  the quality of FMCSA’s compliance review processes. The Secretary of  Transportation tasked the Federal Aviation  Administration, as a peer of  FMCSA within DOT, to conduct a review and develop appropriate  recommendations for DOT’s response to the National Transportation  Safety Board. The Federal Aviation Administration formed an  Independent  Review Team, which in July 2014, issued a report that  included a range of recommendations intended to support both  incremental and transformative improvements to FMCSA’s compliance  and enforcement programs. We discuss some steps FMCSA is taking to  address these recommendations later in this report.
section_title: CSA Interventions
paragraphs: Under the CSA program, FMCSA can select from a range of eight  intervention types, intended to give FMCSA the flexibility to address  motor carriers’ specific safety problems. Four of the intervention types  were newly introduced under the CSA program; FMCSA had been  applying the other four intervention types prior to the program (see   table 3). Each type falls into one of the following intervention categories:  early contact, investigation, and follow-on interventions.
Before the CSA program, FMCSA used one investigation intervention  type—the onsite compliance review—and three follow-on intervention  types. Compliance reviews required investigators to examine every part  of a carrier’s operations and were thus extremely resource intensive to  conduct. As a result, FMCSA and its state partners investigated only  about 3 percent of active carriers. According to FMCSA Motor Carrier  Safety Progress Reports, federal or state investigators conducted  between approximately 15,000 and 17,000 compliance reviews each year  from fiscal year 2006 through fiscal year 2009.
section_title: CSA Intervention Process
paragraphs: Under the CSA program, FMCSA has established a process for  measuring the relative safety risk of carriers in seven Behavioral Analysis  and Safety Improvement Categories (BASICs), prioritizing carriers based  on risk, assigning an appropriate intervention type, and investigating or  enforcing compliance with regulations (see fig. 1). The CSA intervention  process has no set progression, and based on existing guidance, FMCSA  applies one or more interventions depending on the circumstances of  each case. For example, FMCSA may directly assign an onsite  comprehensive investigation to a carrier without first assigning another  type of intervention. FMCSA may also apply multiple interventions to a  carrier over time, and as a result, common patterns in FMCSA’s  application of CSA interventions can be identified. For example, if a  carrier receives a warning letter as a first intervention and its safety  performance does not improve, FMCSA may assign the carrier a second  intervention, such as an onsite focused investigation. If FMCSA identifies  violations during an investigation, such as the onsite focused  investigation, that warrant enforcement, FMCSA may assign a third  intervention, such as a notice of claim. In such a case, the resulting  intervention pattern would be: (warning letter) → (onsite focused  investigation) → (notice of claim).
Every month, FMCSA uses SMS to generate percentiles in any of seven  BASICs for carriers with sufficient data. However, as we reported in  2014, the SMS methodology contains limitations that reduce FMCSA’s  ability to reliably assess carriers’ safety risks because FMCSA lacks  sufficient safety performance information on the majority of carriers. We  identified the lack of sufficient information as a particular limitation for  carriers with few inspections and vehicles because their underlying  violation rates can have artificially low or high values, or greater  variability, which affects the precision of the BASIC percentiles used by  FMCSA in comparing carriers to one another.
Nonetheless, FMCSA uses these BASIC percentiles to identify potentially  risky carriers, to prioritize them for intervention, and to automatically  generate warning letters. FMCSA automatically prioritizes carriers into  risk-based categories of escalating urgency based on the extent to which  a carrier’s BASIC percentiles exceed certain combinations of designated  thresholds, in addition to the carrier’s intervention history and unresolved  violations. For example, under the new high-risk definition FMCSA  adopted in March 2016, for a carrier to be considered high-risk, its BASIC  percentile has to be above 90 in at least two of four BASICs—unsafe  driving, crash indicator, hours-of-service compliance, or vehicle  maintenance—and the carrier cannot have had an onsite comprehensive  investigation in the last 18 months. In addition to changing the high-risk  definition, FMCSA implemented a new prioritization approach with five  risk-based categories, from most to least urgent: high-risk, moderate-risk,  risk, warning letter, and monitor. According to FMCSA, high-risk carriers  are the agency’s highest investigative priority.
After FMCSA prioritizes carriers for intervention, FMCSA division  managers decide which intervention type carriers should receive based on priority level, guidance, and carrier history among other things. The  principal guidance document for assigning intervention types is the  electronic Field Operations Training Manual (eFOTM). Although eFOTM  includes some requirements, division managers have some discretion to  select investigation and follow-on intervention types based on additional  information, such as complaints received, significant crashes, and  professional judgment. For example, FMCSA must investigate non- frivolous written complaints that allege substantial violations regardless of  whether the carrier is prioritized for intervention, but have discretion to  determine the most appropriate investigation type based on the safety  problems associated with the complaint, the carrier’s BASIC percentiles,  and the carrier’s history. However, for carriers identified as high-risk,  FMCSA must conduct an onsite focused or onsite comprehensive  investigation.
Federal Safety Investigators and their state partners follow eFOTM  guidance when conducting investigation and follow-on interventions. In  April 2013, FMCSA introduced enhanced investigative techniques (EIT)  that are intended to help investigators identify the root cause of a motor  carrier’s safety problems. While financial and legal penalties are typically  applied following an investigation, FMCSA may levy financial penalties  against a carrier without an investigation if it believes there is sufficient  evidence, such as evidence that the carrier operated after being placed  out of service. Investigators may also request a change to the intervention  type for some carriers when they find new and pertinent information that  was not available at the time of the assignment.
section_title: FMCSA Information Systems
paragraphs: FMCSA’s information systems are critical to its data-driven enforcement  and compliance programs and are intended to provide real-time access to  data for the enforcement community, the transportation industry,  stakeholders, and the general public. Field staff input intervention data  through a variety of field information systems. These systems are  operated on laptop computers in the field. For example, field staff use the  Compliance Analysis and Performance Review Information (CAPRI)  system to enter investigation intervention data, such as investigatory files  and safety violations identified. Similarly, they use CaseRite to enter legal  enforcement information. As previously discussed, FMCSA plans to  introduce a new field information system called SENTRI, which will  consolidate the legacy information systems that field staff use to upload  information related to interventions.
Once uploaded by field staff, intervention data are stored and analyzed  on multiple central information systems. FMCSA staff may access CSA  intervention data on these systems through a centralized portal and use  the data to monitor carriers’ safety performance, among other things. For  example, the Motor Carrier Management Information System (MCMIS)  includes motor carrier performance data including inspection and  investigation results, enforcement data, and state-reported crashes.  FMCSA also uses the Enforcement Management Information System  (EMIS) to monitor, track, and store data related to FMCSA enforcement  actions, including follow-on interventions. FMCSA’s Analysis and  Information Online system provides public access to descriptive statistics  and analyses regarding commercial vehicle, driver, and carrier safety  information.
section_title: Information Technology Delays Have Hampered Full Implementation of CSA Interventions and Limited the Quality of Available Information
subsections: section_title: FMCSA Has Delayed Implementing Two New CSA Intervention Types Nationwide Until It Develops Information Technology Software
paragraphs: Although FMCSA implemented all four new CSA intervention types in  pilot test states, the agency chose to delay implementing two of the new  intervention types in the remaining states until it develops information  technology (IT) software.
Implementation in Pilot Test States: FMCSA implemented the entire  range of CSA interventions—including all four new CSA intervention  types—in nine pilot test states as part of the Operational Model Test that  FMCSA conducted from February 2008 through June 2010. The test  was intended to help the agency assess the four new intervention types  and identify any features that needed to be adjusted prior to implementing  them nationwide, among other things. According to FMCSA headquarters  officials, personnel experienced challenges using multiple legacy  information systems that were not designed to support FMCSA’s  application of the expanded range of interventions under the CSA  program. For example, FMCSA’s data analysts found it difficult to extract  data from information systems needed to monitor and oversee the  agency’s application of interventions.
Implementation in Non-Pilot Test States: According to headquarters  officials, in July 2010 FMCSA chose to delay implementing two of the four  new CSA intervention types—offsite investigations and cooperative safety  plans—in the remaining non-pilot test states until it completes its  development of SENTRI software. However, FMCSA decided to  implement the remaining two new intervention types—warning letters and  onsite focused investigations—nationwide because they believed those  two interventions were demonstrated as effective during the Operational  Model Test and that delays would hinder safety benefits for the public  (see fig. 2). The Operational Model Test evaluation found that offsite  investigations demonstrated a similar pattern of effectiveness as onsite  focused and onsite comprehensive investigations.
FMCSA headquarters officials told us that developing and implementing  SENTRI is important to help field staff and their state partners conduct  their work. Field staff currently may use a variety of legacy information  systems to apply and manage interventions. Principally, field staff use the  CAPRI system to prepare for investigation interventions and to report  their results. However, CAPRI was designed to support traditional  compliance review investigations, not the expanded range of investigation  types under the CSA program. According to FMCSA officials, this has  resulted in field staff taking time-consuming additional steps to report their  application of interventions. For example, some division administrators  spent additional time reviewing how investigators entered information into  CAPRI to determine the correct investigation type performed. According  to officials, SENTRI is expected to help address these inefficiencies by  consolidating investigative, follow-on, reporting, and other functions into a  single interface. FMCSA officials also expect SENTRI to improve data  consistency and enable better policy and program decisions through  improved data quality.
However, FMCSA has faced longstanding delays in developing  SENTRI  software as part of its broader IT modernization effort. In September  2005, FMCSA initiated a comprehensive overhaul of the way it collects,  manages, and conveys safety information. The agency-wide  modernization effort was intended to help FMCSA achieve its  effectiveness and efficiency outcomes for the CSA program by  centralizing FMCSA data and simplifying information access, among  other things. According to FMCSA headquarters officials, FMCSA began  obligating funds to develop SENTRI software in fiscal year 2009, when it  established the business case for the system (see app. II).  Since that  time, we and the DOT’s Office of Inspector General have reported  continuing project delays.
FMCSA hired consultants to identify the causes of, among other things,  its IT project delays and actions to remediate them. The resulting March  2013 report found a variety of underlying program challenges. For  example, it found that ineffective IT governance practices provided limited  visibility into the health of individual projects, contributing to project  delays. It also found that the lack of an appropriately scoped and  measurable strategy made it unclear whether current resources were  effectively prioritized—a challenge that was compounded when priorities  shifted on multiple occasions over time.
Officials stated that FMCSA executed a contract in January 2016 to  complete the agency’s IT modernization effort. As part of this effort,  FMCSA plans to complete its development of SENTRI by April 2017.
section_title: FMCSA’s Application of CSA Interventions Has Declined, but Data Limitations Affect the Quality of Available Information
subsections: section_title: Declining Intervention Application
paragraphs: FMCSA’s application of interventions declined from fiscal year 2012  through fiscal year 2015, according to estimates provided by the agency.  FMCSA implemented warning letters nationwide in fiscal year 2011,  which resulted in a temporary spike in interventions. However, after this  temporary increase, the number of interventions FMCSA applied was less  in fiscal year 2015 than in fiscal year 2012 for each intervention type, with  notable decreases in offsite investigations (73 percent) and notices of  violation (71 percent). In addition, according to FMCSA’s estimates, about  26 percent fewer total investigation interventions were conducted in fiscal  year 2015 compared to fiscal year 2012. See table 4 for detailed  estimates of FMCSA’s application of interventions. Reasons for these  notable decreases are discussed below.
Offsite investigations: Division officials from each of the four pilot test  states we interviewed told us they selected offsite investigations less  frequently in more recent years, because the number of carriers that met  eFOTM criteria for receiving them decreased over time. For example, officials from one division told us their use of offsite investigations  decreased because motor carriers’ BASIC percentiles were typically too  high or involved too many BASICs to qualify to receive an offsite  investigation according to current eFOTM criteria. FMCSA headquarters  officials told us they are focused on increasing the use of offsite  investigations, because they believe offsite investigations were  demonstrated as both efficient and effective. For example, in March 2016  FMCSA established a working group to explore modifying policy to give  division managers more discretion in assigning offsite investigations.
Notices of violation: Division officials from four of the eight divisions we  interviewed told us they selected notices of violation infrequently because  the intervention type was time-intensive to process compared to other  intervention types or was not appropriate to address severe safety  violations. FMCSA headquarters officials told us that investigators prefer  to issue notices of claim instead, because they result in penalties to motor  carriers. However, officials stated that investigators may not be aware of  other associated FMCSA activities that increase the overall resources  used to issue notices of claim. For example, notices of claim require  additional legal oversight, which generally requires more resources.  Headquarters officials said investigators may choose to issue notices of  violation rather than notices of claim, when appropriate, if they better  understood this context.
Total investigation interventions: FMCSA headquarters officials told us  that total investigation interventions declined because investigators spent  more time conducting in-depth reviews of motor carriers’ safety  management practices to identify the root causes of underlying safety  problems as part of FMCSA’s EIT initiative. FMCSA implemented EIT in  fiscal year 2013 as part of continuous improvement efforts and in  response to Independent Review Team recommendations. According to  officials, using the more time-intensive EIT approach decreased the  number of investigations that FMCSA could conduct, particularly since  2012. FMCSA officials stated investigation counts have increased  somewhat as investigators adjusted to the new EIT procedures. However,  the officials did not expect investigation counts to return to previous levels  without additional personnel.
section_title: Data Quality Limitations
paragraphs: As FMCSA introduced an expanded range of intervention types under the  CSA program, FMCSA did not redesign CAPRI or other legacy  information systems to reflect these changes. For example, FMCSA did  not redesign CAPRI to include a dedicated data element that would  uniquely record the occurrence of each intervention type. Instead  FMCSA developed algorithms—rules that can be applied by computer  programs—that attempted to reconstruct the occurrence of each  intervention type by identifying specific combinations of multiple data  elements.
Using legacy information systems for purposes for which they were not  designed produced two main limitations that affected the accuracy of  FMCSA intervention counts:  Data recording limitations: FMCSA headquarters officials stated that the  accuracy of CSA intervention counts depended in part upon how users  recorded intervention data into FMCSA’s IT systems. For example,  although all but 10 states did not implement offsite investigations, the  CAPRI system nonetheless allowed investigators in these states to select  “offsite” for the “review location” data element. According to FMCSA  officials, this inflated counts when FMCSA used “review location” as one  of multiple data elements to identify offsite investigations. Similarly,  investigators may conduct onsite focused investigations on carriers that  receive complaints, but CAPRI requires investigators to select either  “complaint” or “focused CR” for the “review reason” data element.  Because FMCSA’s algorithm used “review reason” as one of multiple  data elements to count onsite focused investigations, this deflated onsite focused investigation counts when investigators selected “complaint” to  conduct them.
Evaluation Limitations: FMCSA officials told us they occasionally modified  algorithms used to identify the occurrence of intervention types, but  generally did not evaluate how the modifications affected the accuracy of  intervention counts. According to officials, they modified algorithms for a  variety of reasons, such as when the agency changed how it recorded  intervention data. Once modified, FMCSA applied the most recent  algorithm to all previous data—including historical data. For example, in  January 2016, FMCSA removed a redundant data element from the  algorithm used to count investigation interventions, a step that changed  historical intervention counts. FMCSA officials told us they did not know  the extent to which applying the modified algorithm to previous data  affected the accuracy of historical counts, because they generally did not  evaluate the modification’s effect on count accuracy before applying it.  Although the extent of the effect is unknown, even small differences could  limit the ability of FMCSA managers to accurately and effectively monitor  trends in FMCSA’s application of CSA interventions over time.
FMCSA headquarters officials told us that SENTRI is intended to address  the underlying IT challenges that limit the accuracy of CSA intervention  counts by creating a dedicated data element that uniquely records the  occurrence of each intervention type. Developing SENTRI in a timely  manner is particularly critical, because data-driven targeted enforcement  is FMCSA’s primary strategy for meeting its safety goals and further  delays represent missed opportunities for FMCSA to accurately monitor  and improve the CSA program. Moreover, unresolved data limitations  would continue to preclude outside entities, such as auditing entities, from  assessing the integrity of agency information, including the completeness  and accuracy of computer-generated counts.
In May 2016, we initiated a review to determine the extent to which  FMCSA has evaluated the effectiveness of selected IT systems, and to  assess the extent to which FMCSA has implemented an IT governance structure and plan to complete this work by June 2017. In addition, as we  discussed above, FMCSA currently estimates that it will complete  SENTRI development in April 2017. In light of our and FMCSA’s ongoing  work in this area, we are not making a recommendation on this matter in  this report.
section_title: FMCSA Evaluated the Effectiveness and Efficiency of CSA Interventions, but the Evaluations Had Limitations
paragraphs: In its Strategic Plan: Fiscal Years 2015–2018, FMCSA identified the  improved effectiveness and efficiency of CSA interventions as strategic  outcomes. FMCSA has evaluated both of these strategic outcomes, but  its evaluations had limitations. Specifically, FMCSA’s effectiveness  evaluations did not produce sufficiently complete, appropriate, and  accurate information on individual  intervention types because of design  and methodological limitations. Additionally,  FMCSA’s efficiency  evaluation is no longer current, because FMCSA has not taken steps to  update the evaluation’s cost estimates, despite changes in the time and  resources required to conduct CSA interventions.
subsections: section_title: FMCSA’s Effectiveness Evaluations Did Not Produce Sufficiently High- Quality Information
paragraphs: FMCSA’s Strategic Plan: Fiscal Years 2015–2018 identifies improved  effectiveness as a strategic outcome for CSA interventions. According to  FMCSA, the agency conducts regular evaluations to determine how  effectively its programs are achieving their effectiveness and other  intended outcomes. To evaluate the effectiveness of CSA interventions  specifically, FMCSA developed a statistical model intended to annually  evaluate the combined effects of all of its interventions. According to  FMCSA, the model is a revised version of a prior model that FMCSA used  to evaluate the effectiveness of compliance reviews, before the  implementation of the CSA program added new intervention types.  FMCSA has also evaluated intervention effectiveness in other studies, but  according to officials, the new annual model is the agency’s primary  method of assessing intervention effectiveness.
In a January 2015 report, the annual effectiveness model estimated the  combined effect of four CSA intervention types on the crash rates of  carriers in four size groups from fiscal year 2009 through fiscal year  2011. To assess effectiveness, the model estimated the change in  group crash rates before and after carriers received one or more  interventions, compared to the change in a comparison group of carriers  that did not receive an intervention. This design accounted for the effects  of some external factors that also could have influenced group crash  rates, such as broad changes in weather or economic conditions. When  used appropriately, a comparison group design is a key strength of a  model, such as the one used by FMCSA, as is the use of statistical  inference to evaluate the certainty of the model’s results.
Federal standards for internal control state that agencies should use  quality information to determine the extent to which they are achieving  their intended program outcomes. Characteristics of quality information  include complete, appropriate, and accurate information that helps  management make informed decisions and evaluate the entity’s  performance in achieving strategic outcomes. Because, according to  headquarters officials, the annual effectiveness model is the primary  method FMCSA uses to evaluate CSA intervention effectiveness and  FMCSA intends to use the model on a recurring basis, we conducted a  detailed assessment of the model. As discussed below, we identified  several design and methodological limitations in FMCSA’s annual  effectiveness model, including the design of its comparison groups, a design limitation that can impact the quality of information that the model  produces (see app. III  for our complete assessment).
subsections: section_title: Information Completeness
paragraphs: FMCSA’s report concluded that applying at least one intervention reduced  crash rates for three out of four carrier size groups in fiscal year 2009 and  fiscal year 2011 and provided positive safety benefits. The report also  concluded that warning letters independently reduced group crash rates  and increased safety benefits. However, FMCSA’s annual evaluation did  not provide sufficiently complete information on intervention effectiveness  because the evaluation did not assess all intervention types or  intervention patterns that FMCSA commonly applies. Specifically, the  evaluation did not explicitly measure follow-on interventions, including  cooperative safety plans, notices of claim, and notices of violation.  According to the January 2015 report, the evaluation did not include  cooperative safety plans because MCMIS  data for that intervention were  not consistently complete or accurate. FMCSA officials told us that the  model does not specifically exclude notices of claim and notices of  violation, but rather simply does not distinguish between investigations  that result in follow-on interventions and those that do not. According to  FMCSA, the model does not make this distinction because FMCSA  typically applies follow-on interventions, such as notices of claim, within  90 days of conducting an investigation, based on the investigation’s  findings.
We determined that FMCSA’s model cannot analyze such intervention  patterns, because the agency designed it to identify only a carriers’ first intervention in a fiscal year. As a result, FMCSA does not have  information on the unique effectiveness of its specific follow-on  interventions, which are critical for enforcing regulatory compliance.  FMCSA could potentially increase the breadth of analysis for its model by  using a design similar to the evaluation of FMCSA’s Operational Model  Test, conducted by UMTRI  in August 2011. That evaluation identified  common patterns of interventions with sufficient data for analysis and  then estimated each pattern’s effectiveness. The combined estimates  supplemented those for individual  intervention types.
section_title: Information Appropriateness
paragraphs: The annual evaluation does not provide information that appropriately  reflects how FMCSA designed and implemented CSA interventions. The  agency designed CSA interventions to replace the resource-intensive  “one-size-fits-all” compliance review with a range of intervention types  intended to better address safety problems specific to individual carriers.  According to FMCSA’s Strategic Plan: Fiscal Years 2015–2018, the  agency’s strategy to reduce the number of unsafe and high-risk carrier  behaviors is to create and apply appropriate interventions. As we  previously discussed, FMCSA’s application of individual intervention  types depends upon a combination of eFOTM rules and managers’ day- to-day discretion. For example, according to eFOTM, managers generally  assign offsite investigations when a carrier has percentiles above  intervention thresholds in three or fewer BASICs. However, managers  have the discretion to apply an onsite focused investigation instead,  based on the carrier’s circumstances. Additionally, FMCSA may apply  multiple intervention types for the same carrier over time, resulting in  commonly observed intervention patterns. For example, one official told  us that most carriers that FMCSA investigates have, at one time, been  issued a warning letter.
In contrast to the design and implementation of CSA interventions,  FMCSA’s model does not include an assessment of either individual  intervention types or common intervention patterns. Instead, the model  estimates the impact of all interventions combined that were performed during a 12-month period being measured. Recommended practices for  program evaluation recommend that program managers attempt to  separately evaluate multiple types of program activities that seek to  achieve a common outcome–in this case, multiple intervention types that  seek to improve carrier safety performance. Consistent with these  practices, FMCSA’s Strategic Plan: Fiscal Years 2015–2018 calls for the  agency to use data to make smarter day-to-day decisions and to  determine the impact that various rules have on decreasing crashes,  injuries, and fatalities by conducting regular program evaluations and  effectiveness reviews.
Therefore, to provide information appropriate to the design and  implementation of CSA interventions, an evaluation should assess how  effectively each intervention type or common intervention patterns  addressed the safety problems of the carriers that received them. This  specific information could help FMCSA identify the circumstances under  which different types of interventions are effective and help managers  optimize their choice of interventions on a day-to-day basis as the agency  implements the program.
Although officials told us that FMCSA designed the model to measure the  cumulative effect of FMCSA contact with carriers through CSA  interventions and not to analyze individual  intervention types, FMCSA  used the model to draw conclusions about the safety benefits of warning  letters, one of the most common intervention types used according to  FMCSA estimates. Specifically, FMCSA concluded that “this   suggests that the warning letter in and of itself can be an effective tool for  improving motor carrier safety.” However, as accepted practices for designing evaluations explain, quality evaluations should draw  conclusions commensurate with the power of the design. Because  FMCSA did not evaluate the separate effect of warning letters, it lacks  specific analytical evidence to support its conclusion. As a result, FMCSA  lacks quality information needed to estimate how each intervention type,  including warning letters, or common intervention patterns affect motor  carrier safety performance and address carriers’ specific safety problems.
FMCSA headquarters officials stated that they were unsure if it was  possible to measure the effects of individual  intervention types or  common intervention patterns because the quantity of available data may  not be sufficient to produce reliable results. However, as previously  discussed, FMCSA’s August 2011 evaluation of the CSA Operational  Model Test, conducted by UMTRI,  assessed the effectiveness of each  intervention type and common intervention patterns for carriers that  received multiple interventions. An evaluation of these effects was  possible for the Operational Model Test, despite the fact that UMTRI  had  less data available than FMCSA would generate after nationwide  implementation of the CSA program. According to the study, UMTRI  assessed such patterns to provide a more detailed look at the  effectiveness of the interventions, in light of how FMCSA actually applied  them in the field. Furthermore, in March 2016, FMCSA conducted  separate evaluations of how two specific intervention types—onsite  focused and offsite investigations—influenced carriers’ BASIC percentiles  in calendar years 2011 and 2012. By combining multiple years of  intervention data, as FMCSA did in these evaluations, rather than using a  fiscal year construction, which officials told us FMCSA used in the annual  evaluation for administrative reasons, FMCSA could potentially overcome  data sufficiency limitations (see app. III).
FMCSA and independent evaluators have identified a need for this level  of detailed information in the management and implementation of  interventions. For example, an internal FMCSA working group determined  that the agency needed a more detailed understanding of the  effectiveness of onsite focused investigations to empower investigators to  select the most appropriate intervention to change carrier behavior.  Additionally,  in its 2014 assessment, the Independent  Review Team that  DOT tasked with reviewing FMCSA’s compliance review processes  identified a need for FMCSA to perform consistent, detailed, evaluations  of effectiveness by enforcement tool, such as intervention types. Without  detailed evaluations, the team said that FMCSA would be unable to focus  resources on using its most effective tools or to reconfigure tools that are  not meeting the agency’s goals.
Nonetheless, as discussed earlier in this report, FMCSA’s ability to  accurately identify specific intervention types is limited. Accepted  practices for designing evaluations state that quality evaluations should  rely on credible data that are sufficiently free of errors that could lead to  inaccurate conclusions. Taking steps to reliably and accurately identify  each intervention type in the data used to support its evaluations would  help FMCSA conduct evaluations that produce information appropriate to  the design and implementation of CSA interventions.
section_title: Information Accuracy
paragraphs: Uses and Characteristics of Comparison  Groups  A comparison group, in the context of typical  designs for evaluating program effectiveness,  represents what would have happened in the  absence of a program and is used to rule out  alternative explanations for changes in  outcomes. In a truly randomized experiment,  this w ould be the control group. In a quasi- experimental evaluation, like FMCSA’s annual  effectiveness evaluation, w here participants  (i.e., carriers) are not sorted randomly into  groups, the comparison group should be  constructed to be as similar as possible to the  group being influenced by the program (i.e.,  carriers receiving interventions), in order to  draw  strong conclusions about the effects of  the program. The groups should be similar  enough that any difference in outcome can be  plausibly attributed to the intervention type  being evaluated.
FMCSA’s annual and separate effectiveness evaluations had  methodological limitations that limited their ability to accurately attribute  changes in carrier safety behavior solely to interventions. Because of  these limitations, FMCSA may not have accurately accounted for factors  other than FMCSA’s interventions that could be responsible for the  outcomes observed. Most notably, FMCSA did not consistently use a  comparison group design, which compares outcomes among carriers that  did and did not receive interventions, for its effectiveness evaluations.  When FMCSA did use this design, it constructed comparison groups that  did not sufficiently account for external factors that could affect group  crash rates. According to recommended practices for designing program  evaluations, comparison group designs are typical for assessing program  effectiveness, because they can isolate a program’s unique effects when  the comparison groups are sufficiently similar to the groups affected by a  program. See appendix III  for a complete assessment of the limitations  we identified in FMCSA’s annual effectiveness evaluation, along with  accepted practices that could help FMCSA to address them.
FMCSA’s separate evaluations of onsite focused and offsite investigation  effectiveness did not use a comparison group and so, as previously  discussed, did not account for external factors that could have influenced  changes in motor carriers’ BASIC percentiles. Officials stated that they  determined that a comparison group method was not appropriate for  FMCSA’s evaluation of onsite focused investigations because the officials  were concerned that a comparison group would overstate the  effectiveness of onsite focused investigations due to the differing safety  profiles of carriers that receive each intervention type. FMCSA  headquarters officials stated that they chose to use the same  methodology as the onsite focused investigation effectiveness evaluation  for the separate evaluation of offsite investigation interventions.
Although FMCSA did use a comparison group approach in its annual  effectiveness evaluation, we identified limitations with FMCSA’s approach  that affect the model’s ability to accurately attribute changes in crash  rates to interventions. For example, FMCSA’s comparison group was  observed over a different measurement period from the carriers that  received interventions, so that the two groups were not perfectly matched  on broad changes, such as weather and economic changes. By not  matching the measurement period, FMCSA’s use of a comparison group  was limited in its ability to account for external factors, as intended.
Additionally,  in the 2015 evaluation, FMCSA’s comparison groups held  constant the effects of carrier size, but did not hold constant key factors  that could influence intervention outcomes, such as pre-intervention  safety behaviors as measured by regulatory violations or BASIC  percentiles. FMCSA headquarters officials said that comparison groups in  the model accounted only for carrier size because of data limitations and  their concern that accounting for too many additional variables would  reduce the power of the model. Specifically, officials said that FMCSA  does not currently have sufficient data to assess the effectiveness of  agency interventions for motor carriers with different characteristics, such  as the total number of miles a carrier’s vehicles travel per year. However,  FMCSA has previously used the data it has available to hold constant  important factors external to the program and attribute changes in  outcomes to its interventions with greater accuracy (see app. III).  For  example, FMCSA’s August 2011 evaluation of the Operational Model  Test, conducted by UMTRI,  matched groups of carriers that did and did  not receive various types of interventions on several key characteristics,  such as the distributions of pre-intervention crash rates and BASIC  percentiles. Without using a robust comparison group or a similar method,  FMCSA cannot accurately determine whether changes in motor carrier  safety performance are a result of interventions or whether other factors  are responsible.
Without more complete, appropriate, and accurate information on the  effectiveness of individual CSA intervention types, FMCSA lacks  information it needs to make informed decisions and evaluate its  performance in achieving its effectiveness outcome for CSA interventions.  Additionally,  FMCSA’s ability to accurately identify specific intervention  types by analyzing MCMIS  and EMIS data is limited due to the data  limitations which we described earlier in this report. Without taking steps  to address these limitations, FMCSA cannot accurately evaluate how  effectively CSA interventions improve motor carriers’ safety performance.
section_title: FMCSA’s Evaluation of Intervention Efficiency Is Based on Outdated and Non-Representative Cost Information
paragraphs: As with its effectiveness outcome, FMCSA identified improved efficiency  in its Strategic Plan: Fiscal Years 2015–2018 as one of its strategic  outcomes for CSA interventions. We have previously reported that  agencies should measure program efficiency by considering the  relationship between two elements: (1) inputs, such as costs or hours  worked, and (2) desired results, such as a program’s effect on conditions  or behaviors. In the past, FMCSA evaluated the efficiency of CSA  interventions using both of these elements. Specifically, UMTRI’s August  2011 evaluation estimated the average cost of conducting individual  intervention types and measured the effects on carrier safety  performance of those CSA intervention types, as well as common  intervention patterns, in four pilot test states during an 8-month period  from October 2008 through May 2009. Since the UMTRI evaluation,  FMCSA has continued to evaluate the effectiveness of CSA interventions.  However, as we describe below, FMCSA has not taken similar steps to  update its cost information—information FMCSA would need to  understand the relationship between both efficiency elements.
The UMTRI  evaluation developed average cost estimates for each  intervention type by considering four cost variables: labor hours,  government miles traveled, vouchers, and other expenses. The  evaluation studied 920 interventions applied to 586 carriers, with the  number of carriers receiving each intervention type ranging from 6  carriers for notices of violation to 249 carriers for onsite focused  investigations. The evaluation concluded that cooperative safety plans  and notices of violation had the lowest average estimated costs—ranging  from $95 to $118 respectively. Onsite comprehensive investigations and  onsite focused investigations had the highest estimated costs, averaging  $1,038 and $677 respectively.
Since UMTRI conducted its August 2011 efficiency evaluation, FMCSA  has continued to use the results to report and estimate the efficiency of  interventions. For example, in its Budget Estimates: Fiscal Year 2017  report, FMCSA requested $2.5 million and 50 additional Program  Analysts to complete the last phase of the CSA program, including  nationwide implementation of offsite investigations. To support this  request, FMCSA stated that offsite investigations were “extremely  efficient” and specifically cited the evaluation’s cost estimate. Similarly,  FMCSA headquarters officials told us they currently use the UMTRI  evaluation to estimate efficiency and to understand the relative costs of  individual intervention types.
However, cost estimates from UMTRI’s August 2011 evaluation are no  longer current, because the time and resources needed to conduct  interventions has changed and the evaluation’s estimates are not  representative of all states.
The evaluation’s estimates are no longer current because the time  and resources needed to conduct interventions has changed. For  example, in April 2013, FMCSA implemented a substantial change to  the way investigators conduct investigations, called EIT, which is  intended to help identify the root cause of motor carriers’ safety  problems. FMCSA headquarters officials told us that, although using  EIT takes additional time, it results in improved motor carrier safety  performance.
The evaluation’s estimates do not represent costs in all states.
Specifically, the evaluation stated that its estimates pertain only to the  four states studied. Thus the evaluation’s estimates may not  appropriately represent the average costs associated with applying  interventions in the remaining 46 states, the District of Columbia, and  Puerto Rico.
Federal standards for internal control state that agencies should use  quality information to achieve the agency’s intended program outcomes.  Quality information includes information that is current. FMCSA  headquarters officials told us that they have not taken steps to update the  cost estimates from the UMTRI evaluation to determine current resources  used in all states to conduct each intervention type because they believed  that FMCSA policy and guidance were sufficiently well designed to enable  division managers to select the least resource-intensive intervention type  necessary to correct a carrier’s safety problem.
In March 2015, FMCSA’s then-Acting Administrator testified that given  FMCSA’s limited resources relative to the size of the regulated motor  carrier population, it is imperative for FMCSA to apply its resources  efficiently. However, without cost estimates for CSA interventions that  are current and representative of all states, FMCSA lacks information it  needs to understand the most efficient methods of conducting CSA  interventions in all states. Because FMCSA lacks current cost  information, it also cannot evaluate or understand the relationship  between these costs and the effectiveness of CSA interventions.
section_title: FMCSA Took Some Steps Intended to Improve CSA Intervention Outcomes, but Lacks Measures to Monitor Progress
paragraphs: FMCSA has taken steps intended to improve the effectiveness and  efficiency of CSA interventions—strategic outcomes—principally by  establishing a working group to address these issues and implementing  some of the group’s recommendations. However, FMCSA has not  established performance measures to monitor progress toward achieving  its intended efficiency outcome for interventions. Without establishing  measures for both outcomes, FMCSA is and will be limited in its ability to  monitor program performance and to balance these two priorities, as  needed.
subsections: section_title: FMCSA Has Taken Some Steps Intended to Improve the Effectiveness and Efficiency of CSA Interventions
paragraphs: In April 2014, FMCSA formed a Continuous Improvement Working Group  (CIWG) tasked with improving the effectiveness and efficiency of the CSA  program, including CSA interventions. Comprised of FMCSA staff from  divisions, service centers, and headquarters, as well as state partners,  the CIWG’s objective was to assess the agency’s intervention and  prioritization processes and to recommend improvements that increase  program effectiveness and efficiency. To develop its recommendations,  the working group reviewed available intervention data—such as  investigation reports—and assessed current intervention practices by  surveying and interviewing field staff, among other things. In February  2015, the CIWG made 20 recommendations intended to achieve its  effectiveness and efficiency objectives. According to FMCSA officials,  the agency had implemented 12 of these recommendations as of April  2016 and was working to implement the remaining 8 recommendations.  Implemented recommendations include:  Changing FMCSA’s high-risk definition and prioritization criteria: In March  2016, FMCSA adopted a change to its definition of high-risk carriers and  the criteria it used to prioritize carriers for intervention. For example, as  discussed above, for a carrier to be defined as high-risk under FMCSA’s  new criteria, it has to exceed higher percentile thresholds than previously  used in any of at least two of four specified BASICs. According to  FMCSA, this decreases the total number of high-risk carriers but better  identifies carriers at a high-risk for crashes and will allow investigators to  investigate higher-risk carriers sooner, using current resources. However,  FMCSA’s continued reliance on BASIC percentiles supported by  insufficient safety data, which we identified in our 2014 report and  discussed above, will limit its ability to effectively identify and prioritize the  highest-risk carriers for intervention.
Changing criteria to receive warning letters: In January 2016, FMCSA  expanded the criteria it used to determine which motor carriers receive  warning letters in an effort to reach more carriers and, according to  FMCSA, to prevent further non-compliance before a more intensive  intervention type becomes necessary. Specifically, the CIWG  recommended that FMCSA send warning letters to carriers with BASIC  percentiles above the intervention threshold in more BASICs than  previously allowed, while shortening the time period carriers have after  receiving a warning letter to improve their safety performance before  being prioritized for an investigation intervention. The CIWG projected  that implementing the change would increase the number of warning  letters that FMCSA issued by 30 percent; however, the CIWG noted that  issuing a warning letter that is not effective may merely postpone the  eventual necessity of a carrier receiving an investigation intervention and  that issuing more frequent warning letters could dilute their effectiveness.
Establishing intervention quality review procedures: In March 2016,  FMCSA issued two memorandums requiring field staff to use tools that  FMCSA developed to evaluate and improve the quality of some  intervention types. Specifically, FMCSA developed one tool to measure  the extent to which investigators appropriately and accurately conducted  onsite focused and comprehensive investigations and completed required  documentation. Division managers are expected to use the tool to  evaluate a selected sample of investigation reports quarterly, identify  areas of needed improvement, and provide training to improve report  consistency and quality. Similarly, FMCSA developed another tool that measures the extent to which investigators’ enforcement cases for  notices of claim and notices of violation include sufficient documentation  to meet evidentiary requirements. The memorandum requires division  managers to ensure that each notice is reviewed to identify areas for  improvement and ensure that enforcement cases are properly completed.  According to FMCSA officials, FMCSA intends to use the results of these  evaluations to identify training or policy clarifications needed to  continuously improve the application and effectiveness of each  intervention performed.
section_title: FMCSA Is Limited in Its Ability to Monitor Progress toward Achieving CSA Intervention Outcomes Because It Lacks Performance Measures
paragraphs: As previously discussed, FMCSA’s Strategic Plan: Fiscal Years 2015– 2018 identified improved effectiveness and efficiency as strategic  outcomes of CSA interventions. FMCSA headquarters officials told us  that effectiveness and efficiency are complementary outcomes that  FMCSA strives to balance. For example, according to officials, while  using EIT requires more time and decreases the number of investigations  that FMCSA can conduct (i.e., efficiency), it also increases investigation  quality (i.e., effectiveness). Thus, senior FMCSA officials stressed the  importance of considering both effectiveness and efficiency in any set of  measures used to monitor interventions, and stated that without treating  these two outcomes as parts of a whole, FMCSA cannot achieve its goals  for CSA interventions.
FMCSA has established some measures for its effectiveness outcome,  and monitors these measures on an annual and ongoing basis. While we  identified several limitations with the design and methodology FMCSA  used in its effectiveness evaluation above, FMCSA has established a  measure for intervention effectiveness—crash rates—and annually  monitors the agency’s performance against that measure. Officials told us  that FMCSA also monitors effectiveness using investigation outcome  measures, such as violation rates and safety ratings, on an ongoing  basis.
However, FMCSA has not established measures to monitor progress  toward achieving its efficiency outcome. According to headquarters  officials, FMCSA considers the efficiency outcome to include two  dimensions: (1) the number of carriers FMCSA reaches through  interventions and (2) the resources required for FMCSA to conduct  interventions, including the time and travel required to complete  investigations. While officials stated that FMCSA monitors some  information related to efficiency, such as the number of investigations  completed and investigative outcomes, officials acknowledged that  FMCSA has not formally established measures for its efficiency  outcome.
Leading practices for performance management state that agencies  should express outcomes in a measurable form and establish a set of  performance measures that help monitor progress toward achieving each  outcome. Additionally,  our work has shown that agencies should create  a set of performance measures that addresses important dimensions of  program performance and balances competing priorities to increase the  usefulness of performance plans in guiding decisions. While our past  work has identified challenges some federal agencies face developing  and using outcome-based efficiency measures, it has also highlighted the  importance of developing such measures.
Because FMCSA does not have a complete set of measures that reflects  both its effectiveness and efficiency outcomes for CSA interventions,  FMCSA managers lack benchmarks needed to regularly monitor progress  toward achieving the outcomes. FMCSA also lacks information needed to  balance these priorities and guide management decisions about  FMCSA’s application of interventions. FMCSA’s limited resources and the  increase in crashes involving motor carriers in recent years highlight the  importance of ensuring that FMCSA regularly measures and monitors  progress toward achieving both of these strategic outcomes.
section_title: Conclusions
paragraphs: Fatalities involving motor carriers have increased—rising from about  4,200 in 2011 to almost 4,500 in 2015—and interventions can play a  critical role in reversing this troubling trend. FMCSA aims to reduce such  fatalities by using a data-driven approach that identifies and intervenes  with the highest-risk motor carriers early. To monitor its performance,  FMCSA has identified improved effectiveness and efficiency as strategic  outcomes for CSA interventions and has taken some steps to improve  agency performance in these areas. For example, since March 2016,  FMCSA has required field staff to use tools that FMCSA developed to  evaluate and improve the quality of onsite investigations and two follow- on interventions.
However, we identified important limitations in the information that  FMCSA used to evaluate the effectiveness and efficiency of interventions.  For example, FMCSA’s effectiveness evaluations did not produce  sufficiently complete, appropriate, and accurate information on individual  intervention types or common intervention patterns, because of design  and methodology limitations. Although FMCSA officials expressed  concern about potential sample size limitations when evaluating the  effectiveness of individual intervention types or common intervention  patterns, the August 2011 UMTRI  evaluation that FMCSA sponsored as  part of its Operational Model Test demonstrated that such evaluations are  feasible. Similarly, FMCSA uses cost estimates from UMTRI’s evaluation  to understand the efficiency benefits of interventions. However, the  evaluation’s cost estimates are no longer current because the time and  resources needed to conduct interventions have changed and are not  representative of costs nationwide. Without current cost estimates that  are representative of nationwide costs, FMCSA lacks information it needs  to understand the most efficient methods of conducting CSA interventions  and cannot assess the relationship between these costs and intervention  effectiveness.
Moreover, long-standing delays in developing SENTRI software have  compromised the quality of intervention information, thereby limiting  FMCSA’s ability to accurately and effectively monitor trends in its  application of interventions over time and to evaluate intervention  effectiveness. As FMCSA continues its efforts to address data limitations  that affect the accuracy of intervention information, it is important that  FMCSA not delay taking steps to improve how it currently evaluates the  effectiveness and efficiency of CSA interventions by ensuring, for  example, that its annual effectiveness evaluation addresses other  limitations we have identified.
FMCSA has dedicated significant resources to transition from a costly  one-size-fits-all approach to a range of more effective and efficient  interventions. However, without improving the quality of information that  FMCSA uses to evaluate its performance, the agency will continue to lack  the information it needs to determine the extent to which it is achieving  these fundamental programmatic improvements.
In addition, we found that although FMCSA has established some  performance measures for its effectiveness outcomes, the agency has  not established measures to monitor progress toward achieving its  efficiency outcome. FMCSA needs information on all dimensions of its  effectiveness and efficiency outcomes to balance these priorities and  guide management decisions about its application of interventions.
section_title: Recommendations for Executive Action
paragraphs: To determine whether CSA interventions influence motor carrier safety  performance, the Secretary of Transportation should direct the FMCSA  Administrator to:  Identify and implement, as appropriate, methods to evaluate the  effectiveness of individual intervention types or common intervention  patterns to obtain more complete, appropriate, and accurate  information on the effectiveness of interventions in improving motor  carrier safety performance. In identifying and implementing  appropriate methods, FMCSA should incorporate accepted practices  for designing program effectiveness evaluations, including practices  that would enable FMCSA to more confidently attribute changes in  carriers’ safety behavior to CSA interventions.
To understand the efficiency of CSA interventions the Secretary of  Transportation should direct the FMCSA Administrator to:
Update FMCSA’s cost estimates to determine the resources currently  used to conduct individual intervention types and ensure FMCSA has  cost information that is representative of all states.
To enable FMCSA management to monitor the agency’s progress in  achieving its effectiveness and efficiency outcomes for CSA interventions  and balance priorities, the Secretary of Transportation should direct the  FMCSA Administrator to:
Establish and use performance measures to regularly monitor  progress toward both FMCSA’s effectiveness outcome and its  efficiency outcome.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the DOT for review and comment.  DOT provided written comments, which are reprinted in appendix IV. In its  written comments, DOT concurred with our recommendations. DOT also  described actions that FMCSA has taken to improve the CSA program  and noted that CSA interventions have been shown to effectively improve  motor carriers’ safety behavior. As stated in this report, the evaluations  that FMCSA uses to assess intervention effectiveness did not produce  sufficiently complete, appropriate, and accurate information on individual  intervention types because of design and methodological limitations that  limited FMCSA’s ability to accurately attribute changes in carriers’ safety  behavior solely to interventions. We believe that identifying and  implementing appropriate methods to address these limitations will help  to provide FMCSA with information it needs to evaluate its performance in  achieving its effectiveness outcome for CSA interventions.
In addition, DOT provided technical comments on the draft report, which  we incorporated as appropriate.
We are sending copies of this report to relevant congressional  committees, the Secretary of Transportation, and the Administrator of  FMCSA. In addition, the report is available at no charge on GAO’s  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of our report were to examine: (1) the extent to which the  Federal Motor Carrier Safety Administration (FMCSA) has implemented  Compliance, Safety, Accountability (CSA) interventions and how it has  applied them; (2) the extent to which FMCSA has evaluated the  effectiveness and efficiency of CSA interventions; and (3) any steps that  FMCSA has taken to improve and monitor progress toward achieving its  intended outcomes for CSA interventions.
To determine the extent to which FMCSA has implemented CSA program  interventions and how it applied them, we analyzed FMCSA intervention  data from fiscal year 2010 through fiscal year 2015, the most recent fiscal  year for which intervention information was available. Specifically, we  analyzed data from two FMCSA data systems: (1) the Motor Carrier  Management Information System (MCMIS)  and (2) the Enforcement  Management Information System (EMIS). If we determined that MCMIS  and EMIS data were sufficiently reliable, we intended to then analyze  whether there were any notable increases, decreases, or other trends in  FMCSA’s application of interventions—across states, regions, and motor  carrier types (e.g., fleet size). We also intended to determine common  intervention patterns when carriers receive multiple interventions (e.g.,  warning letter, then off-site investigation, then notice of violation).
To determine the reliability of FMCSA data we requested a complete set  of all MCMIS  and EMIS data from fiscal year 2010 through fiscal year  2015. To develop our request, we conducted interviews with cognizant  FMCSA officials as well as officials from the Volpe National  Transportation Systems Center, which provides technical support to  FMCSA’s data analysis. We would typically review documentation to  prepare a data analysis plan; however, FMCSA could not provide up-to- date or complete data dictionaries or other reference documents for these  data systems. Thus, we requested and FMCSA provided sample data  tables and variables that we could use to identify the occurrence of each  intervention type by analyzing MCMIS  and EMIS data.
Using the information that FMCSA provided, we performed electronic  data testing to count how frequently FMCSA conducted each intervention  type in fiscal year 2010 through fiscal year 2015. We then compared the  results of our analysis against FMCSA-published  sources to determine if  the results included obvious errors or outliers. While we replicated  FMCSA’s counts for warning letters and unsatisfactory/unfit out-of-service  orders in all fiscal years, our comparison revealed differences in at least  one fiscal year for all remaining intervention types. We subsequently met  with FMCSA data analysis officials on several occasions to identify the  cause of the differences that we identified. FMCSA officials told us that  when they modified the algorithms used to count interventions, the  agency applied the most recent algorithm to all previous data—including  historical data—thereby changing the way that FMCSA counted  interventions over time. As a result, FMCSA officials told us that we could  not validate the results of our analysis against agency totals.
After evaluating the reliability of these data for our analytical and reporting  purposes, we concluded that the data were of undetermined reliability,  because data limitations prevented an adequate and comprehensive  assessment. This precluded our analysis of trends in FMCSA’s  application of interventions across states, regions, and motor carrier  types. As a substitute, we requested that FMCSA provide estimates for  how frequently it applied each intervention type from fiscal year 2010  through fiscal year 2015 to identify general trends. After reviewing  FMCSA documentation related to the estimates and interviewing  responsible FMCSA officials, we determined that FMCSA’s estimates  were sufficiently reliable for this purpose.
We reviewed relevant regulations and FMCSA guidance and policy  documents to identify how FMCSA should implement and apply  interventions. For example, we reviewed the electronic Field Operations  Training Manual (eFOTM), which is the principal guidance document for  assigning intervention types.
In addition, we interviewed FMCSA division officials in eight states that  we selected based upon their participation in FMCSA’s Operational Model  Test of the CSA program, geographic location, and program size, among  other factors. Selected states included:
Georgia
Illinois
Kansas
Maryland
Massachusetts
Montana
Oklahoma
Texas  We selected these states to get a range of perspectives on FMCSA’s  application of interventions. For example, four of the states participated in  FMCSA’s Operational Model Test and thus had experience implementing  all eight CSA intervention types. Similarly, we selected two states from  each service center. Although the information obtained from our  interviews with officials from the selected states is not generalizable to all  states or FMCSA divisions, it provided illustrative examples of how  FMCSA is applying interventions as well as the perspectives of officials  knowledgeable about the program. In addition, we interviewed FMCSA  officials from each service center—including FMCSA’s Eastern,  Midwestern, Southern, and Western Service Centers—as well as  headquarters officials from FMCSA’s Office of Enforcement, Office of  Field Operations, and Office of Research and Information Technology.  We also interviewed industry stakeholders, such as the Commercial  Vehicle Safety Alliance, American Trucking Associations, Trucking  Alliance, and the Owner-Operator Independent Drivers Association to  gain their perspectives on FMCSA’s intervention and enforcement  activities.
To determine the extent to which FMCSA has evaluated the effectiveness  and efficiency of CSA interventions, we intended to conduct our own  effectiveness evaluation to determine how interventions affect motor  carrier safety and illustrate the strengths and limitations of particular  evaluation designs. However, because MCMIS  and EMIS data were of  undetermined reliability, we instead reviewed the four evaluations the  agency has conducted. Specifically, we reviewed:
The University of Michigan Transportation Research Institute,  Evaluation of the CSA 2010 Operational Model Test (August 2011);
FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier  Intervention Effectiveness Model, Version 1.0: Summary Report for  Fiscal Years 2009, 2010, 2011 (January 2015);
FMCSA, Analysis Brief: Effectiveness of Onsite Focused  Investigations (March 2016); and
FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis  (March 2016)
We conducted a more detailed assessment of the second report on  FMCSA’s annual effectiveness evaluation model because, according to  FMCSA headquarters officials, it is the primary method FMCSA uses to  evaluate intervention effectiveness and FMCSA intends to use it on a  recurring basis. In addition, we reviewed FMCSA policy documents— such as FMCSA’s Strategic Plan: Fiscal Years 2015–2018 and eFOTM  guidance—to determine how FMCSA used the information produced by  each of its four evaluations. We also interviewed FMCSA headquarters  officials responsible for developing policy and conducting data analysis as  well as officials from the Volpe National Transportation Systems Center  responsible for designing and conducting some FMCSA evaluations.
To assess FMCSA’s effectiveness evaluations, we consulted GAO’s  guidance on designing program evaluations, which describes accepted  practices for evaluating program effectiveness based on GAO studies,  policy documents, and program evaluation literature. We also consulted  federal standards for internal control for using quality information, and  drew on internal methodological expertise to assess the extent to which  the designs, implementation, and results of FMCSA’s evaluations met  quality information standards.
To determine any steps that FMCSA has taken to improve and monitor  progress toward achieving its intended outcomes for interventions, we  reviewed relevant FMCSA strategic planning and policy documents that  established such outcomes, principally FMCSA’s Strategic Plan: Fiscal  Years 2015–2018. We reviewed reports from external entities that  included recommendations intended to support improvements to  FMCSA’s compliance and enforcement programs, such as the  Independent  Review Team’s July 2014 report and a March 2014 report  from the Department of Transportation’s Office of Inspector General.
Similarly, we reviewed FMCSA’s Continuous Improvement Working  Group’s February 2015 report that included recommendations intended to  achieve FMCSA’s effectiveness and efficiency outcomes.
We also interviewed responsible division, service center, and  headquarters officials to identify any steps FMCSA has taken to monitor  or improve the effectiveness or efficiency of interventions as well as to  determine their perspectives on the effects of these steps. For example,  we interviewed responsible officials at headquarters who develop policy  and oversee adherence to federal regulations by interstate motor carriers  across the country to understand how they monitor or improve  interventions. We also interviewed service center and division officials to  understand the field’s involvement in FMCSA’s improvement activities.  We compared the results of our documentary review and interviews  against leading practices for performance management as well as key  attributes of successful performance measures identified in our prior body  of work. Although GPRAMA’s requirements apply at the departmental  level (e.g., the Department of Transportation), we have previously stated  they can serve as leading practices at other organizational levels, such as  component agencies, offices, programs, and projects.
We conducted this performance audit from July 2015 to October 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on the audit objectives.
section_title: Appendix II: Obligations to Develop Safety Enforcement Tracking and Investigation System Software
paragraphs: Headquarters officials told us that the Federal Motor Carrier Safety  Administration (FMCSA) obligated funds to develop Safety Enforcement  Tracking and Investigation System (SENTRI) software each year from  fiscal year 2009 through fiscal year 2013; however, they could not  determine the exact amount of funds because FMCSA did not track  information technology investments at the project level during those  years. In addition, the agency obligated about $12 million from fiscal year  2014 through fiscal year 2016 in contractor costs to develop the  Compliance, Safety, Accountability (CSA) component of SENTRI (see  table 5). FMCSA officials told us that FMCSA made some progress in  developing the CSA component of SENTRI as a result of these  investments. For example, FMCSA coordinated with field staff to identify  system requirements.
section_title: Appendix III: Analysis of the Federal Motor Carrier Safety Administration’s Carrier Intervention Effectiveness Model
paragraphs: The Federal Motor Carrier Safety Administration (FMCSA), in conjunction  with the John A. Volpe National Transportation Systems Center (Volpe)  has modified an existing effectiveness model to develop a statistical  model, the Carrier Intervention Effectiveness Model (CIEM), which  annually measures the effectiveness of interventions. In January 2015,  FMCSA published its first report using the CIEM, which evaluated the  effectiveness of compliance reviews, and interventions FMCSA  conducted in fiscal years 2009, 2010, and 2011. We analyzed the CIEM  and the January 2015 evaluation report, using accepted practices for  designing program evaluations and internal staff expertise. Below, we  identify methodological strengths and limitations of these efforts, in  addition to potential methods FMCSA could use to improve the  capabilities of its model to estimate program impacts. Specifically, we  have identified strengths and limitations in four key aspects of FMCSA’s  effectiveness model: (1) general design, (2) comparison group, (3)  observation periods, and (4) statistical analysis and inference.
subsections: section_title: General Design
paragraphs: According to FMCSA documentation, the CIEM is a statistical impact  evaluation model that uses historical data to compare the safety  improvement of carriers receiving FMCSA interventions (the treatment  group) to carriers that do not (the comparison group). The January 2015  evaluation report assessed two intervention types that existed prior to the  Compliance, Safety, Accountability (CSA) program, including compliance  review investigations and Performance and Registration Information  Systems Management letters, and four new CSA intervention types,  including warning letters, offsite investigations, onsite focused  investigations, and onsite comprehensive investigations.
To estimate the impact of these interventions, the CIEM measures the  difference between crash rates among carriers in the treatment group  before and after receiving interventions, and then subtracts the difference  in crash rates among carriers in the comparison group. The comparison  group accounts for confounding factors (other than FMCSA interventions)  that may affect safety performance during the post-intervention period,  such as broad changes in weather or economic conditions.
The model is designed to estimate the impact of interventions carried out  in a single fiscal year and measures crash rates over a 12-month period  following the first intervention a carrier receives in the fiscal year. The  model estimates the combined impact of all interventions performed  during 12-month periods, not the unique impact of each individual  intervention type.
We identified the following limitations:
Lack of process evaluation: The CIEM is designed to evaluate  impact, but does not include a process study. According to accepted  practices for designing program evaluations, a program logic model or  process evaluation that identifies the most important external  influences on desired program outcomes is valuable in planning an  impact evaluation that convincingly rules out most plausible  alternative explanations for the observed results. Process  evaluations clarify the program as implemented and specify which of  its activities may be responsible for the observed outcomes. The CSA  program’s logic model might include elements of the CSA program,  such as FMCSA and state safety inspectors and information systems,  roadside inspections, safety interventions, and crashes, injuries,  deaths, and monetary losses prevented.
Without an initial process evaluation, the impact evaluation cannot  precisely identify what aspects of the program affect safety outcomes,  or whether the estimated impacts reflect the program as designed.  Without studying the program’s implementation and comparing its  theoretical logic model to actual practices, it is uncertain whether  impact estimates represent the effectiveness of the program’s  activities as designed or the activities that program staff happened to  have used in practice. The CIEM’s ability to accurately evaluate the  impact of the CSA program could be improved by taking into account  the program’s strategy and goals, and studying how the program is  implemented.
Exclusion of intervention types: The January 2015 evaluation  report did not include all intervention types. According to the report,  the evaluation did not assess cooperative safety plans or direct  notices of violation and direct notices of claim, because data on these  intervention types had inconsistent completeness and accuracy.  FMCSA officials told us that the evaluation included carriers that  received follow-on notices of violation or claim, but it did not  separately estimate their effects. When notices of violation or claim  follow an investigation, the model implicitly estimates their effects in  the post-intervention crash rate. However, the evaluation could have  excluded some carriers that received a notice as their first intervention  in the modeled year but not in the same fiscal year as the  investigation.
Aggregation of intervention types: The CIEM implicitly estimates  how combinations of interventions affect safety, not the effects of  each individual intervention type. The model identifies only the first  intervention that a carrier receives in the modeled year, and estimates  the intervention’s effect on crash rates from that time of first contact.
According to FMCSA officials, the model is designed to evaluate the  effect of general FMCSA contact with carriers through interventions.  Officials said that they tested some alternatives to using the first  intervention in the fiscal year, including using a carrier’s most severe  or last intervention in the fiscal year. Agency staff told us that they  ultimately preferred to use the first intervention, because it  represented the beginning of FMCSA’s influence on a carrier within  the target time period.
FMCSA did not seek to estimate the effectiveness of individual  intervention types, because agency staff had concerns about the  small amount of data available on specific intervention types and  common intervention patterns when carriers receive multiple  interventions over time. FMCSA officials stated that Volpe conducted  preliminary analysis of the data used for the CIEM to determine  potential sample sizes, but did not provide documentation on the  sample sizes for each intervention type or common patterns of  interventions. The challenges arising from small sample sizes could  potentially be addressed by pooling together data from multiple years,  rather than relying on intervention data from one fiscal year.
Recommended practices of program and policy evaluation  recommend that program managers attempt to separately evaluate  multiple types of program activities that seek to achieve a common  outcome. Such “comparative effectiveness (or efficiency)”  evaluations give managers information on how various activities  perform compared to each other. Variation in outcomes across  settings or populations can be the result of variation in program  operations, such as varying types or levels of enforcement. Further,  variation in outcomes associated with features under program control,  such as different agency activities, may identify opportunities for  managers to take action to improve performance.
To obtain this kind of information on comparative effectiveness, the  treatment effects of interventions could be disaggregated into more  nuanced categories than having at least one intervention. This  approach could directly evaluate how each of several intervention  types, or common intervention patterns, affect safety outcomes.  Detailed performance information would better align with the design of  the CSA program and give staff flexibility to choose from a range of  intervention types that can address each carrier’s unique safety  problems. With evidence of comparative effectiveness and efficiency,  FMCSA would have better information on whether specific CSA  interventions or combinations of interventions are more effective than  others in certain circumstances.
The challenges from small sample sizes on particular interventions  could be addressed in several ways. Pooling together data from  multiple years, rather than relying on intervention data from one fiscal  year, might produce a sufficient sample for evaluating less commonly  used interventions. A multi-year design might become viable as the  CSA program continues to produce data over several years, though  pooling data might increase the potential for unmeasured factors to  influence safety outcomes. A process evaluation, as discussed above,  could clarify how FMCSA field staff and state partners have  implemented the program and could identify intervention types, or  specific combinations of interventions, with sufficient data for analysis.
Regression to the mean: The CIEM design does not fully account for  the possibility that variation over time in the treatment group’s safety  outcomes reflects regression to the mean. Regression to the mean is  a statistical phenomenon in which, following an extreme measurement  assumed to be due to random sources of variation, such as sampling  error, subsequent measurements are likely to be closer to the  average, or mean.
Under the CSA program, FMCSA prioritizes carriers to receive  interventions based on whether the carriers’ percentiles exceed pre- determined thresholds in any of seven behavioral analysis and safety  improvement categories (BASIC). FMCSA’s decision to intervene  with a carrier largely depends on BASIC percentiles, and officials use  these percentiles, in addition to other criteria set out in guidance, to  determine which type of intervention to apply. This is especially true  for carriers that exceed the thresholds intended to identify the highest  risk carriers, because FMCSA policy requires that those carriers  receive onsite investigations.
However, some carriers, especially those with few inspections or  vehicles, may go above the intervention threshold in one  measurement period due to anomalous events, but return below the  threshold in the next measurement period due to factors unrelated to  intervention. For example, a carrier may generally violate vehicle  maintenance regulations at the industry average rate over the long- term. The carrier’s estimated violation rate in FMCSA data—and its  related BASIC percentile—may vary around the long-term average in  any particular period. The degree of variation could reflect the actual  inconsistency of the carrier’s maintenance practices over time or  sampling error in the estimation of its violation rate, related to its  frequency of inspection (exposure to violating regulations). A large  deviation in one period from the long-term average could exceed  BASIC percentile thresholds and trigger additional FMCSA oversight,  but the deviation may not reflect a real change in the carrier’s long- term maintenance behavior. In a subsequent period, the carrier’s  BASIC percentile has a higher probability of returning to the long-term  average than continuing at the extreme from the previous period,  assuming the prior deviation was caused by random sources of  variation, such as sampling error. The same process may apply to  crash rates.
According to recommended practices for evaluating the impact of a  program, the evaluation must be carefully designed to rule out  plausible alternative explanations for the results. The CIEM’s quasi- experiment design implicitly controls for differences across carriers  that do not vary substantially over short time periods, which could  include carrier management practices, leadership, and operating  routes and procedures. The design controls for industry-wide changes  over time that are constant across carriers, changes that could include  weather and economic conditions. Lastly, the design explicitly controls  for carrier size by stratifying the analysis by size groups. Although the  size control may account for differences within carriers over time  among the treatment carriers, the model includes few other controls  that might specifically address this potential threat to valid causal  inference. By not fully accounting for regression to the mean, the  CIEM could be attributing changes in outcomes to CSA interventions,  when those changes would have occurred on their own without  intervention.
Accordingly, there is some evidence that some carriers’ safety  behavior improves without intervention. For example, the Independent  Review Team found that, of the carriers FMCSA prioritized for  intervention in 2013, nearly 33 percent had a BASIC percentile above  threshold when assigned for intervention that was no longer above  threshold at the time of the review, suggesting that carriers’ safety  performance may improve naturally without intervention. As we have  previously reported, carriers with fewer vehicles experience wide  variance in crash and violation rates, which can make them especially  prone to regression to the mean.
An alternative design might compare carriers just above and just  below the intervention-triggering  threshold (at a given point in time).  This “regression discontinuity” design would lend itself better to  interventions triggered automatically when carriers exceed some  threshold and would better reflect the nature of the CSA program, as  recommended in accepted practices for evaluation design.
Another alternative approach that could address the regression to the  mean issue would be to match carriers based on variation over time in  the safety outcomes prior to exceeding a BASIC percentile threshold.  This design would fall into a general class of methods that include  pre-treatment outcomes as an additional covariate. This would  enhance the comparison group’s control for any deviations in the  outcome over time prior to treatment and thereby ensure that  treatment carriers would be matched to comparison carriers with  similar outcome dynamics.
section_title: Comparison Group
paragraphs: The CIEM constructs comparison groups using carriers that did not  receive any of the model’s interventions during the modeled, prior, or  subsequent year. The model assigns carriers to one of several  comparison groups, using the same measure of size—the number of  vehicles—used to assign carriers to treatment groups. These separate  comparison groups are intended to eliminate differences associated with  carrier size from the model’s calculation of adjusted crash rates.
We identified the following limitation:
Limited control: Accepted practices for program evaluation would  classify the CIEM as a “quasi-experimental” design for estimating  impact. Quasi-experimental designs compare outcomes in a treatment  group to outcomes in a comparison group formed using non-random  assignment. Due to the lack of randomized assignment, the treatment  and comparison groups may differ on other factors that affect the  outcome. To compensate for this potential bias, evaluators generally  ensure that such confounding variables are held constant in the  construction of the comparison group or use other methods of  adjustment.
The CIEM explicitly holds constant one factor in the construction of its  comparison group: carrier size. The model implicitly controls for all  factors that vary between the treatment and control groups but remain  constant over time, such as state or region of operation. In addition,  the model implicitly controls for all factors that vary over time and  affect the treatment and comparison groups equivalently,  such as  industry-wide effects due to weather or economic conditions.
The CIEM’s “difference-in-difference” design provides these forms of  implicit control, and thus is a key strength of the model. However,  FMCSA might try to construct a more robust comparison group that  explicitly controls for more than just carrier size. Since the design  controls for variables that are fixed across carriers and vary in the  same ways within groups over time, FMCSA might construct a  comparison group that controls for change across multiple variables. If  reliable data were available, then potential variables could include:  multiple measures of size; state of registration, inspections, or  violations; driver characteristics; pre-treatment safety outcomes; pre- treatment BASIC percentiles; and pre-treatment inspections and  regulatory violations. Controlling for pre-treatment outcomes and  BASIC percentiles would be especially desirable and would address  the potential limitation of regression to the mean because the  fluctuations due to sampling error would be controlled by design.
Data availability  and reliability  may limit FMCSA’s ability to include  additional control variables to construct comparison groups. For  example, the 2015 evaluation report notes that data on vehicle miles  traveled—a measure of carrier exposure to crashes—were less  reliable than vehicle count data because, according to officials, they  were sometimes incomplete or inconsistent across carriers. Similarly,  officials told us that FMCSA considered using carrier operation type to  construct comparison groups, but ultimately did not because some  carriers reported multiple operation types or changed their operation  types over time. Given the central importance of the CSA program to  FMCSA’s enforcement efforts, the agency would benefit from  improving or expanding data collection to support a more robust  model.
section_title: Observation Periods
paragraphs: The CIEM defines different observation periods for the treatment and  control groups. For the treatment group, the CIEM uses the date of the  first intervention in the modeled fiscal year as the demarcation between a  12-month pre-intervention period and a 12-month post-intervention  period. Pre- and post-intervention crash rates are measured for these 12- month periods. For the comparison group, in contrast, the CIEM defines  18-month periods preceding and following the midpoint of the modeled  fiscal year to measure pre- and post-intervention crash rates. (This is  because comparison carriers do not have an intervention date to define  the pre- and post-intervention periods and measure crash rates.) The  evaluation report noted that these longer 18-month periods ensure that  the comparison group’s crash rates cover the entire treatment group time  frame. To adjust for the 50 percent longer observation period for carriers  in the comparison group, the evaluation divided crash rates for those  carriers by 1.5 to yield annual crash rates.
We identified the following limitation:  Inconsistent time periods: According to accepted practices for  program impact evaluation, a design must confidently rule out non- program influences that could cause changes in outcomes to occur.  The CIEM does not completely account for factors that might have  affected crash rates for both treatment and comparison groups,  because the lack of overlap between the measurement time periods  does not hold constant factors unique to the season or period of  measurement. For example, as FMCSA officials noted to us, crash  rates are known to depend on seasonal and periodic changes in  weather, and the model’s lack of overlap in measurement time periods  meant that treatment and comparison groups were subject to different  seasonal and periodic conditions. Other potential confounding  variables include seasonal or periodic variation in economic demand  and state enforcement resources.
The design might use measurement periods that vary as a function of  each observed intervention’s timing and characteristics. For example,  the design might construct a custom comparison group for each  member of the treatment group, selecting multiple comparison carriers  using size and other potential confounding variables. A matching  design in which each treatment carrier is compared to one or more  control carriers would allow identical observation periods while still  producing an impact estimate for interventions applied during a single  fiscal year.
section_title: Statistical Analysis and Inference
paragraphs: The CIEM’s use of statistical inference is appropriate to quantify the  uncertainty of its impact estimates. Carrier behavior and safety outcomes  are consistent with a partially random data generation process. In this  context, statistical inference estimates the sampling variability  of the data  over multiple hypothetical realizations of the same process. Applying  inferential statistical methods appropriately reflects the potential for the  observational regulatory, intervention, and safety data to vary partially at  random.
A critical estimate in the CIEM is the net crash rate change for the  treatment group. The model defines this quantity as the difference  between the treatment group’s pre- and post-intervention crash rates,  after subtracting the crash rate change in the comparison group. The  model then tests whether the net change differs from zero at the 0.05  statistical significance level. The model excludes insignificant findings  from its subsequent estimates of total safety benefits calculated.
The model estimates safety benefits by transforming the estimated net  change in crash rates due to interventions into an estimate of total  crashes avoided, using the treatment group’s pre-intervention crash rate  per vehicle and post-intervention vehicle counts. The model uses  historical crash severity data to further estimate injuries prevented and  lives saved associated with each prevented crash.
The model extrapolates these safety benefits—crashes avoided, injuries  prevented, and lives saved—to carriers that received interventions, but  were excluded from the treatment group due to missing or outlier crash or  vehicle count data, and, according to officials, intrastate carriers that were  excluded from the treatment group in the January 2015 version of the  model. The January 2015 evaluation notes that FMCSA assumed that  these carriers will exhibit the same response to intervention as the  carriers included in the model. Accordingly, the model adds the estimated  safety benefits for carriers included in the model to those for carriers with  outliers and missing data. The sum determines the aggregate estimated  safety benefits. According to the 2015 evaluation report, FMCSA  extrapolated safety benefits to the following number of carriers that  received an intervention but were not included in the model: 9,567  carriers in fiscal year 2009; 9,929 carriers in fiscal year 2010, and; 14,816  carriers in fiscal year 2011.
We identified the following limitations:
Multiple hypothesis tests: The model uses multiple statistical  hypothesis tests to calculate total safety benefits. The CIEM estimates  impact on crash rates within four strata of treatment and comparison  groups defined by carrier size. If the net change in crash rates within  each stratum is statistically distinguishable from zero at the 0.05  confidence level, the model uses the results to estimate total safety  benefits by summing the estimated benefits across groups.  Statistically insignificant results in any stratum provide zero safety  benefits by assumption. In this sense, the model’s estimate of total  safety benefit reflects the results of 4 separate hypothesis tests, each  at the 0.05 confidence level. = 1 – (1 – α)K. When K = 4 and α = 0.05, α = 0.18. See Hayes, 450. methods of hypothesis testing typically recommend adjusting the  confidence level of each individual test, such as by applying  Bonferroni adjustments, so that the group-wise error probability  matches the analyst’s intended risk level for all planned tests. These  adjustments typically produce lower alpha values for each individual  test.
Multiple  hypothesis testing methods would be appropriate for the  CIEM, given that it ultimately seeks to estimate total safety benefits as  a function of multiple hypothesis tests. Without making these  adjustments, the CIEM’s estimates of total safety benefits may have  more risk of error than FMCSA intends to accept because the model  does not conduct a joint test. That is, the probability that at least one  group’s safety benefits equals zero may exceed the 0.05 confidence  level that FMCSA accepts in each individual  test. An alternative  approach might calculate the confidence interval of the summed  safety benefit estimate and test whether it equals zero, consistent with  the discussion below.
Confidence intervals of estimated impacts: The CIEM uses  inferential statistical methods to test the hypothesis that the impact for  each pair of treatment and control groups equals zero. However, the  CIEM does not conduct statistical inference, such as estimating  confidence intervals, when analyzing the net change in crash rates  and associated total safety benefits. This approach is inconsistent,  given that the model’s hypothesis test for a non-zero net change in  crash rates implies that the quantity is a random variable with a  sampling distribution and confidence interval. The model’s authors  agree with this implication, suggesting that the test is equivalent to a  “95 percent confidence interval that does not include zero.” Total  safety benefits must also have a confidence interval, given that it is a  function of the change in net crash rates. Nevertheless, the CIEM  reports only point estimates of safety benefits, without conveying the  statistical uncertainty that the model’s assumptions imply.
Accordingly, the CIEM might estimate and report confidence intervals  for its estimates of crash rate impact and safety benefits, in order to  make the statistical inference consistent and quantify the uncertainty  of its estimates. The current hypothesis testing approach may  produce a point estimate for safety benefits that appears more precise  than the underlying confidence interval would support.
section_title: Appendix IV: Comments from the Department of Transportation
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, H. Brandon Haller (Assistant  Director), William Colwell (Analyst in Charge), Katherine Blair, Melissa  Bodeau, Russell Burnett, David Hooper, Benjamin Licht, Grant Mallie,  Ifunanya Nwokedi, Malika Rice, Sandra Sokol, Niti Tandon, and Jeff  Tessin made key contributions to this report.