MOTOR CARRIERS

Better Information 

Needed to Assess 

Effectiveness and 

Efficiency of Safety 

Interventions

Report to Congressional Committees

October 2016

GAO-17-49

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-17-49, a report to 

congressional committees

October 2016

MOTOR CARRIERS

Better Information Needed to Assess 

Effectiveness and Efficiency of Safety 

Interventions

What GAO Found

In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to 

delay nationwide implementation of two of the eight interventions that FMCSA 

uses to address motor carrier safety concerns under its Compliance, Safety, 

Accountability (CSA) program. This delay is linked to continuing delays in 

developing software needed to support the two interventions, offsite 

investigations and the use of cooperative safety plans. The software under 

development is intended to help FMCSA overcome some of the information 

challenges it faces due to its reliance on legacy information systems. FMCSA

estimates that the software development project will be completed by April 2017. 

FMCSA has conducted evaluations of the effectiveness and efficiency outcomes

it established for the CSA program. However, GAO identified several limitations 

in FMCSA’s approaches that impact the usefulness of the evaluations: 

• Intervention effectiveness: FMCSA has developed a statistical model to 

annually evaluate the combined effectiveness of interventions. Although the 

model has some key strengths, such as accounting for a broad range of 

external factors, GAO identified a number of design and methodology 

limitations that reduce the usefulness of its results. For example, the model 

does not include an assessment of individual intervention types. Without this 

type of specific information, FMCSA is hampered in its ability to identify the 

circumstances under which different types of interventions are effective. 

Similarly, these types of limitations affect FMCSA’s ability to accurately draw 

conclusions about intervention effectiveness across all intervention types. 

• Intervention efficiency: To assess the efficiency of CSA interventions, 

FMCSA has relied on a study that it sponsored and that was published in 

2011. This study estimated the average cost of conducting interventions in 

four states from October 2008 through May 2009. However, FMCSA has not 

taken steps to update its cost estimates for interventions since the 2011 

evaluation, despite changes since that time in the resources needed to 

conduct CSA interventions; nor has it taken steps to develop additional 

information that is representative of the costs in other states. Without current 

cost estimates that are representative of all states, FMCSA cannot

appropriately assess the efficiency of its interventions.

FMCSA has taken some actions to improve the effectiveness and efficiency of 

CSA interventions, but lacks measures to monitor progress. In April 2014, 

FMCSA established a working group to assess CSA interventions and make 

recommendations for improvement. As of April 2016, the group had made 20 

recommendations, of which 12 had been implemented. However, GAO found 

that while FMCSA has established some performance measures for its 

effectiveness outcome that are appropriate, it has not established similar 

measures for its efficiency outcome. FMCSA headquarters officials told GAO that 

effectiveness and efficiency are complementary outcomes that FMCSA strives to 

balance. Without a complete set of measures for both outcomes, FMCSA lacks 

benchmarks needed to regularly measure progress to achieve these outcomes.

View GAO-17-49. For more information, 

contact Susan Fleming at (202) 512-2834 or 

flemings@gao.gov.

Why GAO Did This Study

As part of its mission to reduce 

crashes and fatalities involving large 

commercial trucks and buses, FMCSA 

seeks to use a data-driven approach to 

identify the highest-risk motor carriers 

and address safety problems by 

applying a range of eight CSA program

enforcement tools, called interventions, 

ranging from warning letters to placing 

carriers out of service. 

A provision in a Senate report requires 

GAO to periodically assess FMCSA's 

implementation of the CSA program. 

This report examines the extent to 

which FMCSA has (1) implemented 

CSA interventions, (2) evaluated the 

effectiveness and efficiency of CSA 

interventions, and (3) monitored 

progress toward achieving outcomes.

GAO reviewed FMCSA data and 

documentation on all eight CSA 

intervention types from fiscal years

2010–2015, including FMCSA’s

strategic planning documents, 

guidance, and program evaluations. 

GAO interviewed industry stakeholders 

and FMCSA officials in headquarters, 

in each of FMCSA’s service centers, 

and in eight states selected for their

participation in FMCSA’s CSA pilot 

test, location, and program size, 

among other factors.

What GAO Recommends

GAO recommends that FMCSA

evaluate the effectiveness of individual 

intervention types, update cost 

estimates so that they are current and 

representative of all states, and 

establish complete performance 

measures.

The Department of Transportation 

concurred with all of GAO’s 

recommendations.
Page i GAO-17-49 Motor Carriers

Letter 1

Background 4

Information Technology Delays Have Hampered Full 

Implementation of CSA Interventions and Limited the Quality of 

Available Information 15

FMCSA Evaluated the Effectiveness and Efficiency of CSA 

Interventions, but the Evaluations Had Limitations 24

FMCSA Took Some Steps Intended to Improve CSA Intervention 

Outcomes, but Lacks Measures to Monitor Progress 36

Conclusions 40

Recommendations for Executive Action 41

Agency Comments 42

Appendix I Objectives, Scope, and Methodology 44

Appendix II Obligations to Develop Safety Enforcement Tracking and 

Investigation System Software 49

Appendix III Analysis of the Federal Motor Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model 50

Appendix IV Comments from the Department of Transportation 62

Appendix V GAO Contact and Staff Acknowledgments 63

Tables

Table 1: Federal Motor Carrier Safety Administration’s (FMCSA) 

Budget Authority for Fiscal Years (FY) 2010 through 2016 

(Dollars in Millions) 4

Table 2: Federal Motor Carrier Safety Administration’s (FMCSA) 

Field Operations Compliance and Enforcement Program 

Full-Time Equivalent Staff by Position Type for Fiscal 

Years (FY) 2010 through 2015 5

Contents
Page ii GAO-17-49 Motor Carriers

Table 3: Compliance, Safety, Accountability (CSA) Program’s 

Intervention Descriptions 9

Table 4: Estimated Compliance, Safety, Accountability (CSA) 

Interventions Conducted by the Federal Motor Carrier 

Safety Administration (FMCSA), Fiscal Years (FY) 2010–

2015 20

Table 5: Federal Motor Carrier Safety Administration (FMCSA) 

Obligations to Develop Safety Enforcement Tracking and 

Investigation System (SENTRI) Software, Fiscal Years 

(FY) 2009–2016 (Dollars in Millions) 49

Figures

Figure 1: Federal Motor Carrier Safety Administration’s (FMCSA) 

Compliance, Safety, Accountability Intervention Process 11

Figure 2: Federal Motor Carrier Safety Administration’s (FMCSA) 

Implementation of Eight Compliance, Safety, 

Accountability (CSA) Interventions, by State 17
Page iii GAO-17-49 Motor Carriers

Abbreviations

BASIC Behavioral Analysis and Safety Improvement Category

CAPRI Compliance Analysis and Performance Review Information

CIEM Carrier Intervention Effectiveness Model

CIWG Continuous Improvement Working Group

CSA Compliance, Safety, Accountability 

DOT Department of Transportation

eFOTM electronic Field Operations Training Manual 

EIT enhanced investigative techniques

EMIS Enforcement Management Information System

FMCSA Federal Motor Carrier Safety Administration 

FY fiscal year 

GPRAMA GPRA Modernization Act of 2010 

IT information technology

MCMIS Motor Carrier Management Information System 

SENTRI Safety Enforcement Tracking and Investigation System

SMS Safety Measurement System

UMTRI University of Michigan Transportation Research Institute

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessaryif you wish to reproduce this material separately.
Page 1 GAO-17-49 Motor Carriers

441 G St. N.W.

Washington, DC 20548

October 27, 2016

Congressional Committees

Commercial trucks and buses are vital transportation links connecting 

goods and people with American markets and communities. However, the 

Federal Motor Carrier Safety Administration (FMCSA) reported that 

crashes involving large commercial trucks and buses increased from 

about 131,000 in 2011 to nearly 157,000 in 2015.1 These crashes harm 

the well-being of the people involved and the companies that own 

commercial trucks and buses (motor carriers). Fatalities resulting from 

these crashes have also increased, rising from about 4,200 in 2011 to 

almost 4,500 in 2015.2 Further, FMCSA estimated that crashes involving 

large trucks and buses cost over $110 billion in 2014.3

FMCSA’s Compliance, Safety, Accountability (CSA) program is intended 

to reduce the number of motor carrier crashes by using a data-driven 

approach to identify and intervene with the highest-risk motor carriers. 

When motor carrier safety problems are identified, FMCSA applies a 

range of intervention types—such as sending warning letters, conducting 

investigations, or issuing civil penalties—that attempt to remedy those 

problems early and before crashes occur.

A provision in a Senate Appropriations Committee report required us to 

periodically assess FMCSA’s implementation of the CSA program.4 This 

report examines: (1) the extent to which FMCSA has implemented CSA 

 1

FMCSA, Motor Carrier Safety Progress Reports for December 31, 2013 and March 31, 

2016. Crash data are from FMCSA’s Motor Carrier Management Information System. 

FMCSA progress reports do not adjust crash statistics for exposure. FMCSA typically 

accounts for exposure by using vehicle miles traveled to determine a crash rate; however, 

using vehicles miles traveled in this way has limitations, because vehicle miles traveled 

information is self-reported by motor carriers and many carriers do not report it to FMCSA. 

2

FMCSA, Motor Carrier Safety Progress Reports for December 31, 2013 and March 31, 

2016. FMCSA progress reports do not adjust fatality statistics for exposure.

3

FMCSA, 2016 Pocket Guide to Large Truck and Bus Statistics (Washington, D.C., May 

2016).

4

This provision is contained in the Senate Appropriations Committee Report, S. Rep. No. 

113-182, at 57, accompanying the Transportation, and Housing and Urban Development, 

and Related Agencies Appropriations Bill, 2015, which was eventually included in the 

Consolidated and Further Continuing Appropriations Act, 2015, Pub. L. No. 113-235, 128 

Stat. 2130 (2014). 

Letter
Page 2 GAO-17-49 Motor Carriers

interventions and how it has applied them; (2) the extent to which FMCSA 

has evaluated the effectiveness and efficiency of CSA interventions; and 

(3) any steps that FMCSA has taken to improve and monitor progress 

toward achieving its intended outcomes for CSA interventions.

To determine the extent to which FMCSA has implemented CSA 

interventions and how it has applied them, we analyzed FMCSA 

intervention data from fiscal year 2010 through fiscal year 2015. We 

intended to analyze whether there were any notable increases, 

decreases, or other trends in FMCSA’s application of interventions—

across states, regions, and motor carrier types. However, data limitations 

prevented an adequate and comprehensive assessment of the reliability 

of FMCSA’s intervention data. For example, officials told us FMCSA 

changed the way it counted interventions over time, and as a result, we 

could not validate the results of our analysis against agency totals—a 

step to testing data reliability. As a result, we concluded that FMCSA’s 

intervention data were of undetermined reliability, a factor that precluded 

our analysis of trends in FMCSA’s application of interventions across 

states, regions, and motor carrier types. As a substitute, we requested 

that FMCSA provide estimates for how frequently it applied each 

intervention type from fiscal year 2010 through fiscal year 2015 to identify 

general trends. After reviewing FMCSA documentation related to the 

estimates and interviewing responsible FMCSA officials, we concluded 

that FMCSA’s estimates were sufficiently reliable for this purpose. We 

also reviewed relevant regulations and FMCSA guidance and policy 

documents to identify how FMCSA should implement and apply 

interventions. We interviewed headquarters officials from FMCSA’s Office 

of Enforcement, Office of Field Operations, and its Office of Research and 

Information Technology. FMCSA has 52 divisions that correspond with 

each state, the District of Columbia, and Puerto Rico. One of the four 

service centers oversees each division. We interviewed FMCSA division 

officials in eight states that we selected based upon their participation in 

FMCSA’s Operational Model Test of the CSA program, geographic 

location, and program size, among other factors, and interviewed FMCSA 

officials from each of the four service centers. Although the information 

obtained from our interviews with officials from the selected states is not 

generalizable to all states or FMCSA divisions, it provided illustrative 

examples of how FMCSA is applying interventions as well as the 

perspectives of officials knowledgeable about the program.

To determine the extent to which FMCSA has evaluated the effectiveness 

and efficiency of CSA interventions, we reviewed the four evaluations the 

agency has conducted. These included FMCSA’s January 2015 
Page 3 GAO-17-49 Motor Carriers

evaluation on intervention effectiveness; the University of Michigan 

Transportation Research Institute’s (UMTRI) August 2011 evaluation of 

the Operational Model Test, sponsored by FMCSA; and two effectiveness 

evaluations on individual intervention types that FMCSA conducted in 

March 2016.5 To assess these evaluations, we consulted accepted 

practices for evaluation design and drew upon internal methodological 

expertise.6 We also interviewed FMCSA headquarters officials 

responsible for developing policy and conducting data analysis, as well as 

Volpe National Transportation Systems Center officials responsible for 

designing and conducting some FMCSA evaluations.

To determine any steps that FMCSA has taken to improve and monitor 

progress toward achieving its intended outcomes for interventions, we 

reviewed relevant FMCSA strategic planning and policy documents, such 

as FMCSA’s Strategic Plan: Fiscal Years 2015–2018. We interviewed 

responsible FMCSA division, service center, and headquarters officials to 

identify any monitoring or improvement steps that FMCSA has taken as 

well as to determine their perspectives on the effect of these steps. We 

compared the results of our documentary review and interviews against 

leading practices for performance management identified in our prior 

body of work.7

We conducted this performance audit from July 2015 to October 2016 in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

 5

FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier Intervention 

Effectiveness Model, Version 1.0: Summary Report for Fiscal Years 2009, 2010, 2011

(January 2015); UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 

2011); FMCSA, Analysis Brief: Effectiveness of Onsite Focused Investigations (March 

2016); and FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis (March 

2016).

6

GAO, Designing Evaluations: 2012 revision, GAO-12-208G (Washington, D.C.: January 

2012). This report is based on GAO studies and policy documents and program evaluation 

literature. To ensure the report’s competence and usefulness as a guide for designing 

evaluations, drafts were reviewed by selected GAO, federal and state agency evaluators, 

and evaluation authors and practitioners from consulting firms.

7

GPRA Modernization Act of 2010 (GPRAMA), Pub. L. No. 111-352, §3, 124 Stat. 3866, 

3867 (codified at 31 U.S.C. § 1115(b)(2), (6)). See also, GAO, Agency Performance 

Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers, 

GAO/GGD/AIMD-99-69 (Washington, D.C.: Feb. 26, 1999). 
Page 4 GAO-17-49 Motor Carriers

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on the audit objectives.

FMCSA was established within the Department of Transportation (DOT) 

in January 2000, and is tasked with promoting safe commercial motor 

vehicle operations and reducing large truck and bus crashes, injuries, and 

fatalities. It seeks to achieve this reduction through regulation, 

enforcement, and partnerships with stakeholders, among other activities, 

and with full accountability to the public through transparency, resultsoriented performance measuring, and managing for results. Since fiscal 

year 2010, FMCSA’s total budget authority to conduct these activities has 

remained relatively stable, increasing about 6.5 percent from fiscal year 

2010 through fiscal year 2016 (see table 1). Funding for implementing 

and applying interventions is included in both the Motor Carrier Safety 

Operations and Programs and Safety Grants budget authorities.

Table 1: Federal Motor Carrier Safety Administration’s (FMCSA) Budget Authority for Fiscal Years (FY) 2010 through 2016 

(Dollars in Millions)

Budget authoritya FY 2010b FY 2011c FY 2012b FY 2013c FY 2014d FY 2015c FY 2016d

Motor Carrier Safety 

Operations & Programs $239.8 $244.1 $244.1 $251.0 $259.0 $259.0 $267.4

Motor Carrier Safety 

Grants $305.4 $307.0 $306.0 $310.0 $313.0 $313.0 $313.0

Total $545.2 $551.1 $550.1 $561.0 $572.0 $572.0 $580.4

Source: FMCSA. | GAO-17-49

a

Budget authority is authority provided by federal law to enter into financial obligations that w ill result 

in immediate or future outlays involving federal government funds. The basic forms of budget 

authority include (1) appropriations, (2) borrowing authority, (3) contract authority, and (4) authority to 

obligate and expend offsetting receipts and collections.

b

Actual budget authority less rescission. 

c

Actual budget authority. 

d

Enacted budget authority. 

Background

FMCSA
Page 5 GAO-17-49 Motor Carriers

The vast majority of FMCSA’s staff are located in field offices, including 

divisions and service centers.8 Field staff are primarily responsible for 

implementing FMCSA’s compliance and enforcement activities, including 

investigations. Federal Safety Investigators represent the majority of 

FMCSA’s compliance and enforcement program staff (see table 2). In 

addition, FMCSA partners with state agencies to perform some 

intervention activities, such as conducting carrier investigations; however, 

FMCSA is responsible for ensuring that commercial motor carriers under 

its authority comply with federal safety regulations.9 FMCSA’s Office of 

Enforcement is the primary office responsible for developing policy for 

FMCSA’s compliance and enforcement program, and overseeing the 

implementation of intervention activities.

Table 2: Federal Motor Carrier Safety Administration’s (FMCSA) Field Operations Compliance and Enforcement Program FullTime Equivalent Staff by Position Type for Fiscal Years (FY) 2010 through 2015

Position Type FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 FY 2015

Safety Investigatorsa 322 317 313 308 302 303

Program Analysts and 

Specialistsb 67 68 78 75 85 101

Management 63 62 61 65 65 62

Administration and 

Technical Supportc 78 73 75 74 81 82

Total 530 520 527 522 533 548

Source: FMCSA. | GAO-17-49

Note: Counts include staff involved in intervention and enforcement activities and thus exclude the 

follow ing positions: state program managers, Border Inspectors, Border Auditors, Transportation 

Specialists, New Entrant managers, New Entrant specialists, New Entrant clerks, Commercial 

Driver’s License staff, and Medical staff.

a

According to FMCSA, the decrease in Safety Investigators since fiscal year 2010 was due to 

turnover and budgetary delays impacting FMCSA’s hiring ability.

b

According to FMCSA, Program Analysts were contractor positions from fiscal year 2010 through 

fiscal year 2015 and thus not included in counts during most of that period. As contractors, the 

positions increased from 7 in fiscal year 2010 to 26 in fiscal year 2015. FMCSA’s count for full-time 

equivalent staff in fiscal year 2015 includes 7 Program Analysts.

c

According to FMCSA, contractors provided additional regional administrative support from fiscal year 

2010 through fiscal year 2015; how ever contractors are not included in the counts.

 8

As noted above, FMCSA has 52 divisions that correspond with each state, the District of 

Columbia, and Puerto Rico. Each division is overseen by one of four service centers.

9

State agencies include state highway patrols, departments of transportation, departments 

of motor vehicles, and public utility commissions. FMCSA oversees all interstate 

commercial motor carriers and all placarded hazardous materials motor carriers 

regardless of inter- or intrastate operations. 
Page 6 GAO-17-49 Motor Carriers

The CSA program is intended to improve the effectiveness of FMCSA’s 

compliance and enforcement programs, while more efficiently using its 

resources to reach carriers that pose the highest safety risk. The CSA 

program has three key components: (1) the Safety Measurement System 

(SMS), meant to identify high-risk carriers by using data from roadside 

inspections and crashes; (2) interventions, which are intended to help 

carriers address safety problems; and (3) the Safety Fitness 

Determination rule.10 In contrast to the one investigation intervention, the 

compliance review, available under FMCSA’s previous approach, FMCSA 

expects the CSA program to more effectively change unsafe behaviors by 

reaching and intervening with more potentially unsafe carriers earlier 

using a range of intervention types to enforce compliance with safety 

regulations. In 2014, we reported on the effectiveness of the SMS 

component of the CSA program.11

FMCSA conducted an Operational Model Test of the program from 

February 2008 through June 2010 in nine pilot states.12 In December 

2010, FMCSA began implementing the CSA program in three phases: (1) 

implementation of SMS and some intervention types nationwide, (2) 

 10In January 2016, FMCSA released a notice of proposed rulemaking on Carrier Safety 

Fitness Determination and published it to the Federal Register for public comment. 81 

Fed. Reg. 3561 (Jan. 21, 2016).

11GAO, Federal Motor Carrier Safety: Modifying the Compliance, Safety, Accountability 

Program Would Improve the Ability to Identify High Risk Carriers, GAO-14-114 

(Washington, D.C.: Feb. 3, 2014). Specifically, we examined the effectiveness of 

FMCSA’s SMS system in assessing safety risk, and reported that FMCSA faces 

challenges in reliably assessing safety risk for the majority of carriers. We recommended 

that (1) FMCSA revise the SMS methodology to better account for limitations in drawing 

comparisons of safety performance information across carriers and, in doing so, conduct a 

formal analysis that identifies the limitations in the data used to calculate SMS scores, 

including variability in the carrier population and the quality and quantity of data available 

for carrier safety performance assessments and limitations in the resulting SMS scores, 

and (2) FMCSA ensure that any determination of a carrier’s fitness to operate accounts for 

the limitations we identified. FMCSA did not concur with our recommendation but stated 

that it would analyze our recommendations as the agency continued to improve the 

effectiveness of the Safety Measurement System. As of October 2016, FMCSA has not 

implemented our recommendations. 

12FMCSA’s Operational Model Test began in February 2008 with four pilot test states 

(Colorado, Georgia, Missouri, and New Jersey) and later expanded to five more states 

(Minnesota, Montana, Kansas, Maryland, and Delaware). For purposes of this report, we 

subsequently refer to those nine states that participated as “pilot test states.” As part of 

the Operational Model Test, FMCSA sponsored and the University of Michigan 

Transportation Research Institute (UMTRI) conducted an August 2011 evaluation of 

intervention effectiveness and efficiency. See UMTRI, Evaluation of the CSA 2010 

Operational Model Test (August 2011).

The CSA Program
Page 7 GAO-17-49 Motor Carriers

introduction of new investigative techniques and the Safety Management 

Cycle, and (3) nationwide rollout of all intervention types and FMCSA’s 

new investigative software, the Safety Enforcement Tracking and 

Investigation System (SENTRI).13

After a series of four commercial motor vehicle crashes—two involving 

buses and two involving trucks—that together resulted in 25 deaths and 

injuries to 83 people, the National Transportation Safety Board 

investigated and, in November 2013, made recommendations to improve 

the quality of FMCSA’s compliance review processes. The Secretary of 

Transportation tasked the Federal Aviation Administration, as a peer of 

FMCSA within DOT, to conduct a review and develop appropriate 

recommendations for DOT’s response to the National Transportation 

Safety Board. The Federal Aviation Administration formed an 

Independent Review Team, which in July 2014, issued a report that 

included a range of recommendations intended to support both 

incremental and transformative improvements to FMCSA’s compliance 

and enforcement programs.14 We discuss some steps FMCSA is taking to 

address these recommendations later in this report.

Under the CSA program, FMCSA can select from a range of eight 

intervention types, intended to give FMCSA the flexibility to address 

motor carriers’ specific safety problems. Four of the intervention types 

were newly introduced under the CSA program; FMCSA had been 

applying the other four intervention types prior to the program (see 

 13The Safety Management Cycle is a tool that FMCSA uses to systematically assess, 

identify, and address motor carrier safety and compliance issues.

14Independent Review Team, Blueprint for Safety Leadership: Aligning Enforcement and 

Risk a report prepared at the request of the Secretary of Transportation to review the 

Federal Motor Carrier Safety Administration’s Safety Oversight of the Motor Carrier 

Industry (July 15, 2014). 

CSA Interventions
Page 8 GAO-17-49 Motor Carriers

table 3).15 Each type falls into one of the following intervention categories: 

early contact, investigation, and follow-on interventions.

 15FMCSA officials told us that some version of the onsite focused investigation also 

existed before CSA, but that it was only used as a follow-up to a compliance review. Thus, 

FMCSA considers the onsite focused investigation to be new under CSA. In addition to 

interventions, FMCSA may apply other enforcement actions to improve safety behavior. 

For example, to avoid further enforcement proceedings a carrier may negotiate a 

settlement agreement that may result in a deferral of, or reduction in, penalties, and a 

carrier may negotiate a consent agreement that will conditionally upgrade its safety rating, 

each is aimed at addressing the root causes of safety problems.FMCSA has issued a 

small but growing number of consent agreements each fiscal year—growing from 2 in 

fiscal year 2010 to 55 in fiscal year 2015. FMCSA is testing an automated system to 

monitor the agency’s use of these agreements in Massachusetts; until the system is fully 

implemented, FMCSA service centers will continue to monitor consent agreements 

manually.
Page 9 GAO-17-49 Motor Carriers

Table 3: Compliance, Safety, Accountability (CSA) Program’s Intervention Descriptions

Intervention Description

New

intervention type

introduced under

the CSA program

Early Contact Interventiona

Warning Letter Letter sent to the motor carrier’s place of business that specifically identifies the 

safety problem and outlines possible consequences of not addressing it. 



Investigation Interventions

Offsite Investigation Investigation conducted remotely using documents submitted by the motor carrier, 

such as toll receipts, border crossing records, or drug testing records.



Onsite Focused 

Investigation

Investigation conducted at the motor carrier’s place of business. Targets specific 

problem areas.



Onsite Comprehensive 

Investigation

Investigation conducted at the motor carrier’s place of business. Addresses all 

aspects of the carrier’s operation, similar to the previously conducted compliance 

review.

Follow-on Interventionsb

Cooperative Safety Plan A voluntary safety improvement plan collaboratively created by the carrier and 

FMCSA, and implemented by the carrier.



Notice of Violation A formal notice of safety problems sent to the carrier that requires a response and 

evidence of corrective action from the carrier to avoid further intervention. 

Notice of Claim The charging of civil penalties against the carrier.

Out-of-Service Order An order requiring the carrier to cease all motor vehicle operations. Includes, but is 

not limited to:

Imminent Hazard: When FMCSA determines that a carrier poses an imminent 

hazard to safety.

Failure-to-pay: When a carrier fails to pay a civil penalty within 90 days.

Unsatisfactory/Unfit: When a carrier has been rated unsatisfactory and prohibited 

from operating.

Source: FMCSA. | GAO-17-49

a

Early contact interventions also include carrier access to safety data and targeted roadside 

inspections; how ever, these are not w ithin the scope of our review.

b

A follow -on intervention, also referred to as an enforcement action, is generally applied follow ing an 

investigation intervention, as a result of the findings of that investigation. Under the CSA program, 

notices of violation and notices of claim can, in some circumstances, be issued to a carrier w ithout a 

preceding investigation. These are referred to as direct notices of violation and direct notices of claim.

Before the CSA program, FMCSA used one investigation intervention 

type—the onsite compliance review—and three follow-on intervention 

types. Compliance reviews required investigators to examine every part 

of a carrier’s operations and were thus extremely resource intensive to 

conduct. As a result, FMCSA and its state partners investigated only 

about 3 percent of active carriers. According to FMCSA Motor Carrier 

Safety Progress Reports, federal or state investigators conducted 
Page 10 GAO-17-49 Motor Carriers

between approximately 15,000 and 17,000 compliance reviews each year 

from fiscal year 2006 through fiscal year 2009.

Under the CSA program, FMCSA has established a process for 

measuring the relative safety risk of carriers in seven Behavioral Analysis 

and Safety Improvement Categories (BASICs), prioritizing carriers based 

on risk, assigning an appropriate intervention type, and investigating or 

enforcing compliance with regulations (see fig. 1). The CSA intervention 

process has no set progression, and based on existing guidance, FMCSA 

applies one or more interventions depending on the circumstances of 

each case. For example, FMCSA may directly assign an onsite 

comprehensive investigation to a carrier without first assigning another 

type of intervention. FMCSA may also apply multiple interventions to a 

carrier over time, and as a result, common patterns in FMCSA’s 

application of CSA interventions can be identified. For example, if a 

carrier receives a warning letter as a first intervention and its safety 

performance does not improve, FMCSA may assign the carrier a second 

intervention, such as an onsite focused investigation. If FMCSA identifies 

violations during an investigation, such as the onsite focused 

investigation, that warrant enforcement, FMCSA may assign a third 

intervention, such as a notice of claim. In such a case, the resulting 

intervention pattern would be: (warning letter) → (onsite focused 

investigation) → (notice of claim).16

 16UMTRI’s August 2011 evaluation of the Operational Model Test identified a range of 

common intervention patterns as FMCSA had actually applied them in the field. The 

evaluation noted that there is no fixed sequence of interventions that must be followed and 

that the CSA program permits the selected intervention to be tailored to the circumstances 

of each case. See UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 

2011) 50. 

CSA Intervention Process
Page 11 GAO-17-49 Motor Carriers

Figure 1: Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety, Accountability Intervention Process

Every month, FMCSA uses SMS to generate percentiles in any of seven 

BASICs for carriers with sufficient data.17 However, as we reported in 

2014, the SMS methodology contains limitations that reduce FMCSA’s 

ability to reliably assess carriers’ safety risks because FMCSA lacks 

sufficient safety performance information on the majority of carriers.18 We 

identified the lack of sufficient information as a particular limitation for 

carriers with few inspections and vehicles because their underlying 

violation rates can have artificially low or high values, or greater 

 17The seven BASICs are (1) crash indicator, (2) controlled substances/alcohol, (3) driver 

fitness, (4) hours of service compliance, (5) hazardous materials compliance, (6) unsafe 

driving, and (7) vehicle maintenance. Depending on the BASIC, carriers generally receive 

percentiles if they meet minimum thresholds of exposure (i.e., number of vehicles or 

inspections), or a minimum number of inspections with violations. For example, as we 

reported in February 2014, FMCSA reported that SMS generated a percentile in at least 

one BASIC for approximately 17.5 percent of active carriers (92,000 of 525,000).

GAO-14-114.

18GAO-14-114. 
Page 12 GAO-17-49 Motor Carriers

variability, which affects the precision of the BASIC percentiles used by 

FMCSA in comparing carriers to one another.19

Nonetheless, FMCSA uses these BASIC percentiles to identify potentially 

risky carriers, to prioritize them for intervention, and to automatically 

generate warning letters. FMCSA automatically prioritizes carriers into 

risk-based categories of escalating urgency based on the extent to which 

a carrier’s BASIC percentiles exceed certain combinations of designated 

thresholds, in addition to the carrier’s intervention history and unresolved 

violations. For example, under the new high-risk definition FMCSA 

adopted in March 2016, for a carrier to be considered high-risk, its BASIC 

percentile has to be above 90 in at least two of four BASICs—unsafe 

driving, crash indicator, hours-of-service compliance, or vehicle 

maintenance—and the carrier cannot have had an onsite comprehensive 

investigation in the last 18 months.20 In addition to changing the high-risk 

definition, FMCSA implemented a new prioritization approach with five 

risk-based categories, from most to least urgent: high-risk, moderate-risk, 

risk, warning letter, and monitor.21 According to FMCSA, high-risk carriers 

are the agency’s highest investigative priority.22

After FMCSA prioritizes carriers for intervention, FMCSA division 

managers decide which intervention type carriers should receive based 

 19As we reported in 2014, because BASIC percentiles are calculated by ranking carriers in 

relation to one another, imprecise rate estimates for some carriers can cause other 

carriers’ BASIC percentiles to be higher or lower than they would be if they were ranked 

against only carriers with more reliable violation rates. This creates the likelihood that 

many BASIC percentiles do not represent a precise safety assessment for a carrier. See 

GAO-14-114.

20Compared to its previous high-risk definition, FMCSA increased the percentile threshold 

for high-risk carriers and simplified the combination of safety categories in which a carrier 

has to exceed the intervention threshold. Under its new definition, to be considered highrisk, a carrier has to exceed the 90th percentile of at least two of the four designated 

categories for 2 consecutive months. Passenger carriers have more stringent criteria, and 

are considered high-risk if they exceed the 90th percentile of at least two of the four 

BASICs for 1 month, and have not had an onsite comprehensive investigation in the last 

12 months. FMCSA announced its changes to the high-risk definition on March 7, 2016, in 

the Federal Register. 81 Fed. Reg. 11875 (Mar. 7, 2016).

21FMCSA’s information systems automatically generate a warning letter for carriers when 

they are placed on the warning letter list.

22Section of 5305(a) of the recently enacted Fixing America’s Surface Transportation Act, 

Public Law 114–94 129 Stat. 1312, 1544 (2015) requires that FMCSA ensure, at a 

minimum, that a review is conducted on motor carriers that demonstrate through 

performance data that they are among the highest-risk carriers for 4 consecutive months. 
Page 13 GAO-17-49 Motor Carriers 

on priority level, guidance, and carrier history among other things. The 

principal guidance document for assigning intervention types is the 

electronic Field Operations Training Manual (eFOTM). Although eFOTM 

includes some requirements, division managers have some discretion to 

select investigation and follow-on intervention types based on additional 

information, such as complaints received, significant crashes, and 

professional judgment. For example, FMCSA must investigate nonfrivolous written complaints that allege substantial violations regardless of 

whether the carrier is prioritized for intervention, but have discretion to 

determine the most appropriate investigation type based on the safety 

problems associated with the complaint, the carrier’s BASIC percentiles, 

and the carrier’s history.23 However, for carriers identified as high-risk, 

FMCSA must conduct an onsite focused or onsite comprehensive 

investigation. 

Federal Safety Investigators and their state partners follow eFOTM 

guidance when conducting investigation and follow-on interventions. In 

April 2013, FMCSA introduced enhanced investigative techniques (EIT) 

that are intended to help investigators identify the root cause of a motor 

carrier’s safety problems. While financial and legal penalties are typically 

applied following an investigation, FMCSA may levy financial penalties 

against a carrier without an investigation if it believes there is sufficient 

evidence, such as evidence that the carrier operated after being placed 

out of service. Investigators may also request a change to the intervention 

type for some carriers when they find new and pertinent information that 

was not available at the time of the assignment. 

FMCSA’s information systems are critical to its data-driven enforcement 

and compliance programs and are intended to provide real-time access to 

data for the enforcement community, the transportation industry, 

stakeholders, and the general public. Field staff input intervention data 

through a variety of field information systems. These systems are 

operated on laptop computers in the field. For example, field staff use the 

Compliance Analysis and Performance Review Information (CAPRI) 

 

23eFOTM includes guidance for implementing this statutory and regulatory requirement. 49 

U.S.C. §31143 and 49 C.F.R. § 386.12. Any person alleging that a substantial violation is 

occurring or has occurred must file a written complaint with FMCSA stating the substance 

of the alleged substantial violation no later than 90 days after the event. 49 C.F.R. § 

386.12(a)(1). The written complaint must be filed with the National Consumer Complaint 

Database at http://nccdb.fmcsa.dot.gov or any FMCSA Division Administrator. 

FMCSA Information 

Systems 
Page 14 GAO-17-49 Motor Carriers

system to enter investigation intervention data, such as investigatory files 

and safety violations identified. Similarly, they use CaseRite to enter legal 

enforcement information. As previously discussed, FMCSA plans to 

introduce a new field information system called SENTRI, which will 

consolidate the legacy information systems that field staff use to upload 

information related to interventions.24

Once uploaded by field staff, intervention data are stored and analyzed 

on multiple central information systems. FMCSA staff may access CSA 

intervention data on these systems through a centralized portal and use 

the data to monitor carriers’ safety performance, among other things. For 

example, the Motor Carrier Management Information System (MCMIS) 

includes motor carrier performance data including inspection and 

investigation results, enforcement data, and state-reported crashes. 

FMCSA also uses the Enforcement Management Information System 

(EMIS) to monitor, track, and store data related to FMCSA enforcement 

actions, including follow-on interventions. FMCSA’s Analysis and 

Information Online system provides public access to descriptive statistics 

and analyses regarding commercial vehicle, driver, and carrier safety 

information.

 24FMCSA officials told us that SENTRI involves two primary components, the New Entrant 

Program component and the CSA component. FMCSA implemented SENTRI’s New 

Entrant Program component, which monitors drivers and carriers during their first 18 

months on the road by conducting a Safety Audit to ensure that new carriers have 

essential safety management practices in place. If new carriers pass the Safety Audit and 

18-month on-road performance period, they graduate and continue to be monitored under 

the CSA program. The New Entrant Program falls outside the scope of our review. Thus, 

we discuss the CSA component of SENTRI in the remainder of this report.
Page 15 GAO-17-49 Motor Carriers

Although FMCSA implemented all four new CSA intervention types in 

pilot test states, the agency chose to delay implementing two of the new 

intervention types in the remaining states until it develops information 

technology (IT) software.

Implementation in Pilot Test States: FMCSA implemented the entire 

range of CSA interventions—including all four new CSA intervention 

types—in nine pilot test states as part of the Operational Model Test that 

FMCSA conducted from February 2008 through June 2010.25 The test 

was intended to help the agency assess the four new intervention types 

and identify any features that needed to be adjusted prior to implementing 

them nationwide, among other things. According to FMCSA headquarters 

officials, personnel experienced challenges using multiple legacy 

information systems that were not designed to support FMCSA’s 

application of the expanded range of interventions under the CSA 

program. For example, FMCSA’s data analysts found it difficult to extract 

data from information systems needed to monitor and oversee the 

agency’s application of interventions.

Implementation in Non-Pilot Test States: According to headquarters 

officials, in July 2010 FMCSA chose to delay implementing two of the four 

new CSA intervention types—offsite investigations and cooperative safety 

plans—in the remaining non-pilot test states until it completes its 

development of SENTRI software. However, FMCSA decided to 

 25In addition to the nine pilot test states, FMCSA subsequently implemented all 

intervention types in Alaska.

Information 

Technology Delays 

Have Hampered Full 

Implementation of 

CSA Interventions 

and Limited the 

Quality of Available 

Information

FMCSA Has Delayed 

Implementing Two New 

CSA Intervention Types 

Nationwide Until It 

Develops Information 

Technology Software
Page 16 GAO-17-49 Motor Carriers

implement the remaining two new intervention types—warning letters and 

onsite focused investigations—nationwide because they believed those 

two interventions were demonstrated as effective during the Operational 

Model Test and that delays would hinder safety benefits for the public 

(see fig. 2). The Operational Model Test evaluation found that offsite 

investigations demonstrated a similar pattern of effectiveness as onsite 

focused and onsite comprehensive investigations.26

 26As discussed later in this report, the evaluation studied 920 interventions applied to 586 

carriers in four pilot test states during an 8-month period from October 2008 through May 

2009.
Page 17 GAO-17-49 Motor Carriers

Figure 2: Federal Motor Carrier Safety Administration’s (FMCSA) Implementation of Eight Compliance, Safety, Accountability 

(CSA) Interventions, by State

FMCSA headquarters officials told us that developing and implementing 

SENTRI is important to help field staff and their state partners conduct 

their work. Field staff currently may use a variety of legacy information 

systems to apply and manage interventions. Principally, field staff use the 

CAPRI system to prepare for investigation interventions and to report 

their results. However, CAPRI was designed to support traditional 

compliance review investigations, not the expanded range of investigation 

types under the CSA program. According to FMCSA officials, this has 

resulted in field staff taking time-consuming additional steps to report their 

application of interventions. For example, some division administrators 

spent additional time reviewing how investigators entered information into 
Page 18 GAO-17-49 Motor Carriers

CAPRI to determine the correct investigation type performed. According 

to officials, SENTRI is expected to help address these inefficiencies by 

consolidating investigative, follow-on, reporting, and other functions into a 

single interface. FMCSA officials also expect SENTRI to improve data 

consistency and enable better policy and program decisions through 

improved data quality.

However, FMCSA has faced longstanding delays in developing SENTRI 

software as part of its broader IT modernization effort. In September 

2005, FMCSA initiated a comprehensive overhaul of the way it collects, 

manages, and conveys safety information. The agency-wide 

modernization effort was intended to help FMCSA achieve its 

effectiveness and efficiency outcomes for the CSA program by 

centralizing FMCSA data and simplifying information access, among 

other things. According to FMCSA headquarters officials, FMCSA began 

obligating funds to develop SENTRI software in fiscal year 2009, when it 

established the business case for the system (see app. II). Since that 

time, we and the DOT’s Office of Inspector General have reported 

continuing project delays.27

FMCSA hired consultants to identify the causes of, among other things, 

its IT project delays and actions to remediate them. The resulting March 

2013 report found a variety of underlying program challenges.28 For 

example, it found that ineffective IT governance practices provided limited 

visibility into the health of individual projects, contributing to project 

delays. It also found that the lack of an appropriately scoped and 

measurable strategy made it unclear whether current resources were 

effectively prioritized—a challenge that was compounded when priorities 

shifted on multiple occasions over time.

Officials stated that FMCSA executed a contract in January 2016 to 

complete the agency’s IT modernization effort. As part of this effort, 

FMCSA plans to complete its development of SENTRI by April 2017.

 27GAO, Motor Carrier Safety: More Assessment and Transparency Could Enhance 

Benefits of New Oversight Program, GAO-11-858 (Washington, D.C.: Sept. 29, 2011); and 

Department of Transportation, Office of Inspector General Audit Report, Actions Are 

Needed to Strengthen FMCSA’s Compliance, Safety, Accountability Program, Report 

Number MH-2014-032 (Mar. 5, 2014).

28FMCSA, Root Cause Analysis: Findings from the Independent Review of the Application 

& Information Systems Modernization (AISM) Program (Mar. 1, 2013). 
Page 19 GAO-17-49 Motor Carriers

FMCSA’s application of interventions declined from fiscal year 2012 

through fiscal year 2015, according to estimates provided by the agency. 

FMCSA implemented warning letters nationwide in fiscal year 2011, 

which resulted in a temporary spike in interventions. However, after this 

temporary increase, the number of interventions FMCSA applied was less 

in fiscal year 2015 than in fiscal year 2012 for each intervention type, with 

notable decreases in offsite investigations (73 percent) and notices of 

violation (71 percent). In addition, according to FMCSA’s estimates, about 

26 percent fewer total investigation interventions were conducted in fiscal 

year 2015 compared to fiscal year 2012. See table 4 for detailed 

estimates of FMCSA’s application of interventions. Reasons for these 

notable decreases are discussed below.

FMCSA’s Application of 

CSA Interventions Has 

Declined, but Data 

Limitations Affect the 

Quality of Available 

Information

Declining Intervention 

Application
Page 20 GAO-17-49 Motor Carriers

Table 4: Estimated Compliance, Safety, Accountability (CSA) Interventions Conducted by the Federal Motor Carrier Safety 

Administration (FMCSA), Fiscal Years (FY) 2010–2015 

Estimated interventions conducted

FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 FY 2015

Early Contact Intervention

Warning Lettera 5,908 39,180 24,126 20,480 20,546 20,463

Investigation Interventions

Offsite Investigationb 715 645 627 634 381 168

Onsite Focused Investigation 1,199 6,246 10,488 9,512 7,376 7,909

Onsite Comprehensive Investigation / 

Compliance Review 15,393 9,661 7,033 5,783 5,890 5,395

Total Investigation Interventionsc 17,307 16,552 18,148 15,929 13,647 13,472

Follow-on Interventionsd

Cooperative Safety Planb 517 627 450 283 277 238

Notice of Violation 94 138 206 115 86 59

Notice of Claim 5,827 5,580 5,972 4,976 4,584 5,066

Out-of-Service Ordere 1,858 1,800 1,889 1,874 1,937 1,760

Source: FMCSA. | GAO-17-49

a

According to FMCSA, full-scale national deployment of w arning letters occurred during fiscal year

2011 resulting in a spike in w arning letters issued.

b

Estimates reflect implementation of offsite investigations and cooperative safety plans only in the 

follow ing states: Alaska, Colorado, Delaw are, Georgia, Kansas, Maryland, Minnesota, Missouri, 

Montana, and New Jersey.

c

Total investigation interventions are the sum of offsite investigations, onsite focused investigations, 

and onsite comprehensive investigations/compliance reviews. According to the Department of 

Transportation, estimates do not include other carrier review types that FMCSA conducts.

d

Follow -on interventions are not totaled because, according to FMCSA officials, issuing follow-on 

interventions depends upon carriers’ compliance and is generally outside the control of FMCSA.

e

Estimates include failure-to-pay orders—when a carrier fails to pay a civil penalty w ithin 90 days—

and Unsatisfactory/Unfit orders, when a carrier has been rated unsatisfactory and prohibited from 

operating. Estimates do not include Imminent Hazard Out-of-Service orders, when FMCSA 

determines that a carrier poses an imminent hazard to safety.

Offsite investigations: Division officials from each of the four pilot test 

states we interviewed told us they selected offsite investigations less 

frequently in more recent years, because the number of carriers that met 

eFOTM criteria for receiving them decreased over time.29 For example, 

 29eFOTM guidance states that offsite Investigations are generally performed when a 

carrier has three or fewer BASICs with percentiles above the intervention threshold. As we 

discuss in appendix I, we concluded that FMCSA data were of undetermined reliability, 

which precluded our analysis of trends in FMCSA’s application of interventions, including 

whether fewer carriers met eFOTM criteria to receive offsite investigations over time. 
Page 21 GAO-17-49 Motor Carriers

officials from one division told us their use of offsite investigations 

decreased because motor carriers’ BASIC percentiles were typically too 

high or involved too many BASICs to qualify to receive an offsite 

investigation according to current eFOTM criteria. FMCSA headquarters 

officials told us they are focused on increasing the use of offsite 

investigations, because they believe offsite investigations were 

demonstrated as both efficient and effective. For example, in March 2016 

FMCSA established a working group to explore modifying policy to give 

division managers more discretion in assigning offsite investigations.

Notices of violation: Division officials from four of the eight divisions we 

interviewed told us they selected notices of violation infrequently because 

the intervention type was time-intensive to process compared to other 

intervention types or was not appropriate to address severe safety 

violations. FMCSA headquarters officials told us that investigators prefer 

to issue notices of claim instead, because they result in penalties to motor 

carriers. However, officials stated that investigators may not be aware of 

other associated FMCSA activities that increase the overall resources 

used to issue notices of claim. For example, notices of claim require 

additional legal oversight, which generally requires more resources. 

Headquarters officials said investigators may choose to issue notices of 

violation rather than notices of claim, when appropriate, if they better 

understood this context.

Total investigation interventions: FMCSA headquarters officials told us 

that total investigation interventions declined because investigators spent 

more time conducting in-depth reviews of motor carriers’ safety 

management practices to identify the root causes of underlying safety 

problems as part of FMCSA’s EIT initiative. FMCSA implemented EIT in 

fiscal year 2013 as part of continuous improvement efforts and in 

response to Independent Review Team recommendations. According to 

officials, using the more time-intensive EIT approach decreased the 

number of investigations that FMCSA could conduct, particularly since 

2012. FMCSA officials stated investigation counts have increased 

somewhat as investigators adjusted to the new EIT procedures. However, 
Page 22 GAO-17-49 Motor Carriers

the officials did not expect investigation counts to return to previous levels 

without additional personnel.30

As FMCSA introduced an expanded range of intervention types under the 

CSA program, FMCSA did not redesign CAPRI or other legacy 

information systems to reflect these changes. For example, FMCSA did 

not redesign CAPRI to include a dedicated data element that would 

uniquely record the occurrence of each intervention type.31 Instead 

FMCSA developed algorithms—rules that can be applied by computer 

programs—that attempted to reconstruct the occurrence of each 

intervention type by identifying specific combinations of multiple data 

elements.

Using legacy information systems for purposes for which they were not 

designed produced two main limitations that affected the accuracy of 

FMCSA intervention counts:32

Data recording limitations: FMCSA headquarters officials stated that the 

accuracy of CSA intervention counts depended in part upon how users 

recorded intervention data into FMCSA’s IT systems. For example, 

although all but 10 states did not implement offsite investigations, the 

CAPRI system nonetheless allowed investigators in these states to select 

“offsite” for the “review location” data element. According to FMCSA 

officials, this inflated counts when FMCSA used “review location” as one 

of multiple data elements to identify offsite investigations.33 Similarly, 

investigators may conduct onsite focused investigations on carriers that 

receive complaints, but CAPRI requires investigators to select either 

“complaint” or “focused CR” for the “review reason” data element. 

Because FMCSA’s algorithm used “review reason” as one of multiple 

data elements to count onsite focused investigations, this deflated onsite 

 30In July 2015, FMCSA developed a draft Field Staffing Model to help headquarters 

estimate the number and distribution of full-time equivalent staff needed to achieve 

agency goals. For example, the model is expected to provide estimates of required federal 

full-time equivalent staff for each position based on defined workload parameters and 

metrics. 

31Data elements are basic units of information that cannot be further subdivided. For 

example, data elements may include ‘City,’ ‘State,’ and ‘Zip Code.’

32We did not assess FMCSA data for individual motor carriers; however, these limitations 

could also affect the accuracy of carrier-level data. 

33In April 2016, FMCSA officials told us the agency modified its algorithm to count only 

offsite investigations conducted within the 10 states that perform offsite investigations. 

Data Quality Limitations
Page 23 GAO-17-49 Motor Carriers

focused investigation counts when investigators selected “complaint” to 

conduct them.

Evaluation Limitations: FMCSA officials told us they occasionally modified 

algorithms used to identify the occurrence of intervention types, but 

generally did not evaluate how the modifications affected the accuracy of 

intervention counts. According to officials, they modified algorithms for a 

variety of reasons, such as when the agency changed how it recorded 

intervention data. Once modified, FMCSA applied the most recent 

algorithm to all previous data—including historical data. For example, in 

January 2016, FMCSA removed a redundant data element from the 

algorithm used to count investigation interventions, a step that changed 

historical intervention counts.34 FMCSA officials told us they did not know 

the extent to which applying the modified algorithm to previous data 

affected the accuracy of historical counts, because they generally did not 

evaluate the modification’s effect on count accuracy before applying it. 

Although the extent of the effect is unknown, even small differences could 

limit the ability of FMCSA managers to accurately and effectively monitor 

trends in FMCSA’s application of CSA interventions over time.

FMCSA headquarters officials told us that SENTRI is intended to address 

the underlying IT challenges that limit the accuracy of CSA intervention 

counts by creating a dedicated data element that uniquely records the 

occurrence of each intervention type. Developing SENTRI in a timely 

manner is particularly critical, because data-driven targeted enforcement 

is FMCSA’s primary strategy for meeting its safety goals and further 

delays represent missed opportunities for FMCSA to accurately monitor 

and improve the CSA program. Moreover, unresolved data limitations 

would continue to preclude outside entities, such as auditing entities, from 

assessing the integrity of agency information, including the completeness 

and accuracy of computer-generated counts.

In May 2016, we initiated a review to determine the extent to which 

FMCSA has evaluated the effectiveness of selected IT systems, and to 

assess the extent to which FMCSA has implemented an IT governance 

 34FMCSA officials told us that the data element became redundant when FMCSA retired 

underlying data tables that served as a “bridge” between data systems before and after 

CSA implementation. To estimate how this change affected historical counts, we 

compared investigation intervention counts for fiscal year 2012 across FMCSA’s Analysis 

and Information Online system’s December 2015 and February 2016 reports. We found, 

for example, that FMCSA’s counts for onsite focused investigations in the February 2016 

report differed by 172 when compared against the counts in the December 2015 report.
Page 24 GAO-17-49 Motor Carriers

structure and plan to complete this work by June 2017. In addition, as we 

discussed above, FMCSA currently estimates that it will complete 

SENTRI development in April 2017. In light of our and FMCSA’s ongoing 

work in this area, we are not making a recommendation on this matter in 

this report.

In its Strategic Plan: Fiscal Years 2015–2018, FMCSA identified the 

improved effectiveness and efficiency of CSA interventions as strategic 

outcomes. FMCSA has evaluated both of these strategic outcomes, but 

its evaluations had limitations. Specifically, FMCSA’s effectiveness 

evaluations did not produce sufficiently complete, appropriate, and 

accurate information on individual intervention types because of design 

and methodological limitations. Additionally, FMCSA’s efficiency 

evaluation is no longer current, because FMCSA has not taken steps to 

update the evaluation’s cost estimates, despite changes in the time and 

resources required to conduct CSA interventions.

FMCSA’s Strategic Plan: Fiscal Years 2015–2018 identifies improved 

effectiveness as a strategic outcome for CSA interventions. According to 

FMCSA, the agency conducts regular evaluations to determine how 

effectively its programs are achieving their effectiveness and other 

intended outcomes. To evaluate the effectiveness of CSA interventions 

specifically, FMCSA developed a statistical model intended to annually 

evaluate the combined effects of all of its interventions.35 According to 

FMCSA, the model is a revised version of a prior model that FMCSA used 

to evaluate the effectiveness of compliance reviews, before the 

implementation of the CSA program added new intervention types. 

FMCSA has also evaluated intervention effectiveness in other studies, but 

according to officials, the new annual model is the agency’s primary 

method of assessing intervention effectiveness.36

 35FMCSA intends to conduct its effectiveness evaluation each year. However, as of 

August 2016, FMCSA has published only its January 2015 report, the first application of 

the model. Officials told us that FMCSA has completed the analysis for its next report, but 

had not yet published it because it was under internal review.

36UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 2011); FMCSA, 

Analysis Brief: Effectiveness of Onsite Focused Investigations (March 2016); and FMCSA, 

Effectiveness of Offsite Investigations Preliminary Analysis (March 2016). 

FMCSA Evaluated 

the Effectiveness and 

Efficiency of CSA 

Interventions, but the 

Evaluations Had 

Limitations

FMCSA’s Effectiveness 

Evaluations Did Not 

Produce Sufficiently HighQuality Information
Page 25 GAO-17-49 Motor Carriers

In a January 2015 report, the annual effectiveness model estimated the 

combined effect of four CSA intervention types on the crash rates of 

carriers in four size groups from fiscal year 2009 through fiscal year

2011.37 To assess effectiveness, the model estimated the change in 

group crash rates before and after carriers received one or more 

interventions, compared to the change in a comparison group of carriers 

that did not receive an intervention. This design accounted for the effects 

of some external factors that also could have influenced group crash 

rates, such as broad changes in weather or economic conditions.38 When 

used appropriately, a comparison group design is a key strength of a 

model, such as the one used by FMCSA, as is the use of statistical 

inference to evaluate the certainty of the model’s results.

Federal standards for internal control state that agencies should use 

quality information to determine the extent to which they are achieving 

their intended program outcomes.39 Characteristics of quality information 

include complete, appropriate, and accurate information that helps 

management make informed decisions and evaluate the entity’s 

performance in achieving strategic outcomes. Because, according to 

headquarters officials, the annual effectiveness model is the primary 

method FMCSA uses to evaluate CSA intervention effectiveness and 

FMCSA intends to use the model on a recurring basis, we conducted a 

detailed assessment of the model. As discussed below, we identified 

several design and methodological limitations in FMCSA’s annual 

effectiveness model, including the design of its comparison groups, a 

 37FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier Intervention 

Effectiveness Model, Version 1.0: Summary Report for Fiscal Years 2009, 2010, 2011

(January 2015). Because the January 2015 report assessed interventions from fiscal 

years 2009, 2010, and 2011, it included some intervention types FMCSA used before 

CSA’s implementation in 2010. Specifically, the report included three types of compliance 

reviews, non-ratable compliance reviews on interstate carriers, and Performance and 

Registration Information Systems Management warning letters. However, these 

intervention types are outside the scope of our review. Carrier size groups are determined 

based on the carrier’s number of vehicles.

38The model estimated how the group crash rate changed for carriers that received at 

least one intervention, and subtracted the change for carriers that did not receive an 

intervention. The model made estimates separately for each of four groups of carriers with 

varying numbers of vehicles (1-5, 6-20, 21-100, and 100 or more). The model estimated 

crash rates by dividing the total number of crashes by the total number of vehicles in the 

12 months before and after the intervention respectively. The model did not analyze the 

crash rates of individual carriers. See appendix III for more information.

39GAO, Standards for Internal Control in the Federal Government, GAO-14-704G 

(Washington, D.C.: September 2014).
Page 26 GAO-17-49 Motor Carriers

design limitation that can impact the quality of information that the model 

produces (see app. III for our complete assessment).

FMCSA’s report concluded that applying at least one intervention reduced 

crash rates for three out of four carrier size groups in fiscal year 2009 and 

fiscal year 2011 and provided positive safety benefits.40 The report also 

concluded that warning letters independently reduced group crash rates 

and increased safety benefits. However, FMCSA’s annual evaluation did 

not provide sufficiently complete information on intervention effectiveness 

because the evaluation did not assess all intervention types or 

intervention patterns that FMCSA commonly applies. Specifically, the 

evaluation did not explicitly measure follow-on interventions, including 

cooperative safety plans, notices of claim, and notices of violation.41

According to the January 2015 report, the evaluation did not include 

cooperative safety plans because MCMIS data for that intervention were 

not consistently complete or accurate. FMCSA officials told us that the 

model does not specifically exclude notices of claim and notices of 

violation, but rather simply does not distinguish between investigations 

that result in follow-on interventions and those that do not. According to 

FMCSA, the model does not make this distinction because FMCSA 

typically applies follow-on interventions, such as notices of claim, within 

90 days of conducting an investigation, based on the investigation’s 

findings.

We determined that FMCSA’s model cannot analyze such intervention 

patterns, because the agency designed it to identify only a carriers’ first 

 40The report concluded that the combined effect of the four intervention types included in 

the model reduced carrier crash rates.Specifically, the model estimated that for all three 

fiscal years, crash rates for carriers with 1-5 vehicles reduced by between 28.8 and 34.5 

percent, and crash rates for carriers with 6-20 vehicles reduced by between 13.9 and 30 

percent. For carriers with 21-100 vehicles, the model estimated that crash rates reduced 

by 7.2 percent in fiscal year 2009 and 15.9 percent in fiscal year 2011, but results were 

not statistically significant in fiscal year 2010. The interventions did not have statistically 

significant effects for carriers in the largest size group in any year. The model defined 

safety benefits as crashes avoided, injuries prevented, and lives saved. FMCSA estimated 

these benefits using the estimated effects on crash rates and historical data on crashes 

involving injuries and fatalities. See appendix III for more information. 

41The model also did not measure out-of-service orders, direct notices of violation, or 

direct notices of claim. Direct notices of violation and claim differ from follow-on notices of 

violation and follow-on notices of claim because they are not issued as the result of an 

investigation. Like cooperative safety plans, the January 2015 evaluation report stated that 

it did not include direct notices of violation or direct notices of claim because MCMIS data 

were not consistently complete or accurate. Officials also told us that these intervention 

types were used too infrequently to be included.

Information Completeness
Page 27 GAO-17-49 Motor Carriers

intervention in a fiscal year. As a result, FMCSA does not have 

information on the unique effectiveness of its specific follow-on 

interventions, which are critical for enforcing regulatory compliance. 

FMCSA could potentially increase the breadth of analysis for its model by 

using a design similar to the evaluation of FMCSA’s Operational Model 

Test, conducted by UMTRI in August 2011. That evaluation identified 

common patterns of interventions with sufficient data for analysis and 

then estimated each pattern’s effectiveness. The combined estimates 

supplemented those for individual intervention types.42

The annual evaluation does not provide information that appropriately 

reflects how FMCSA designed and implemented CSA interventions. The 

agency designed CSA interventions to replace the resource-intensive 

“one-size-fits-all” compliance review with a range of intervention types 

intended to better address safety problems specific to individual carriers. 

According to FMCSA’s Strategic Plan: Fiscal Years 2015–2018, the 

agency’s strategy to reduce the number of unsafe and high-risk carrier 

behaviors is to create and apply appropriate interventions. As we 

previously discussed, FMCSA’s application of individual intervention 

types depends upon a combination of eFOTM rules and managers’ dayto-day discretion. For example, according to eFOTM, managers generally 

assign offsite investigations when a carrier has percentiles above 

intervention thresholds in three or fewer BASICs. However, managers 

have the discretion to apply an onsite focused investigation instead, 

based on the carrier’s circumstances. Additionally, FMCSA may apply 

multiple intervention types for the same carrier over time, resulting in 

commonly observed intervention patterns. For example, one official told 

us that most carriers that FMCSA investigates have, at one time, been 

issued a warning letter.

In contrast to the design and implementation of CSA interventions, 

FMCSA’s model does not include an assessment of either individual 

intervention types or common intervention patterns. Instead, the model 

estimates the impact of all interventions combined that were performed 

 42UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 2011),49-56 and 

63-75. UMTRI’s evaluation estimated the effectiveness of interventions FMCSA applied 

during its Operational Model Test in four pilot test states (Colorado, Georgia, Missouri, 

and New Jersey) from February 2008 through June 2010. It estimated the effectiveness of 

individual interventions and the most frequently observed intervention patterns by

comparing BASIC intervention thresholds exceeded for carriers that received interventions 

to three groups of control carriers. The report noted that evaluating intervention patterns 

was important, because they were fundamental to the CSA process. 

Information Appropriateness
Page 28 GAO-17-49 Motor Carriers

during a 12-month period being measured. Recommended practices for 

program evaluation recommend that program managers attempt to 

separately evaluate multiple types of program activities that seek to 

achieve a common outcome–in this case, multiple intervention types that 

seek to improve carrier safety performance.43 Consistent with these 

practices, FMCSA’s Strategic Plan: Fiscal Years 2015–2018 calls for the 

agency to use data to make smarter day-to-day decisions and to 

determine the impact that various rules have on decreasing crashes, 

injuries, and fatalities by conducting regular program evaluations and 

effectiveness reviews.

Therefore, to provide information appropriate to the design and 

implementation of CSA interventions, an evaluation should assess how 

effectively each intervention type or common intervention patterns 

addressed the safety problems of the carriers that received them. This 

specific information could help FMCSA identify the circumstances under 

which different types of interventions are effective and help managers 

optimize their choice of interventions on a day-to-day basis as the agency 

implements the program.

Although officials told us that FMCSA designed the model to measure the 

cumulative effect of FMCSA contact with carriers through CSA 

interventions and not to analyze individual intervention types, FMCSA 

used the model to draw conclusions about the safety benefits of warning 

letters, one of the most common intervention types used according to 

FMCSA estimates. Specifically, FMCSA concluded that “this [analysis] 

suggests that the warning letter in and of itself can be an effective tool for 

improving motor carrier safety.”44 However, as accepted practices for 

 43See, for example, Anna R. Solmeyer and Nicole Constance, “Unpacking the ‘Black Box’ 

of Social Programs and Policies: Introduction,” American Journal of Evaluation 36 (4), 

470-474 and related articles in the edited journal volume.

44FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier Intervention 

Effectiveness Model, Version 1.0: Summary Report for Fiscal Years 2009, 2010, 2011

(January 2015). FMCSA’s January 2015 effectiveness evaluation report stated that since 

the vast majority of warning letters were not followed by an intervention in the same fiscal 

year, the results of implementing the model for carriers with warning letters as the first 

intervention were likely to be similar to what would be obtained by considering carriers that 

received warning letters only. The evaluation report did not include any analysis to support 

this assumption; however, FMCSA provided us with some information that indicated that 

between 3.1 and 12.8 percent of carriers that received a warning letter received a 

subsequent intervention in the same fiscal year during fiscal years 2009 through 2011. 

Because the model tracks each carrier for a full 12 months after its first intervention, the 

percentages of carriers receiving subsequent interventions during the model’s 

measurement period would be higher than these. 
Page 29 GAO-17-49 Motor Carriers

designing evaluations explain, quality evaluations should draw 

conclusions commensurate with the power of the design.45 Because 

FMCSA did not evaluate the separate effect of warning letters, it lacks 

specific analytical evidence to support its conclusion. As a result, FMCSA 

lacks quality information needed to estimate how each intervention type, 

including warning letters, or common intervention patterns affect motor 

carrier safety performance and address carriers’ specific safety problems.

FMCSA headquarters officials stated that they were unsure if it was 

possible to measure the effects of individual intervention types or 

common intervention patterns because the quantity of available data may 

not be sufficient to produce reliable results.46 However, as previously 

discussed, FMCSA’s August 2011 evaluation of the CSA Operational 

Model Test, conducted by UMTRI, assessed the effectiveness of each 

intervention type and common intervention patterns for carriers that 

received multiple interventions. An evaluation of these effects was 

possible for the Operational Model Test, despite the fact that UMTRI had 

less data available than FMCSA would generate after nationwide 

implementation of the CSA program. According to the study, UMTRI 

assessed such patterns to provide a more detailed look at the 

effectiveness of the interventions, in light of how FMCSA actually applied 

them in the field.47 Furthermore, in March 2016, FMCSA conducted 

separate evaluations of how two specific intervention types—onsite 

focused and offsite investigations—influenced carriers’ BASIC percentiles 

 45For more information on accepted practices for evaluating program effectiveness see 

GAO, Designing Evaluations: 2012 revision, GAO-12-208G (Washington, D.C.: January 

2012). This report is based on GAO studies and policy documents and program evaluation 

literature. To ensure the guide’s competence and usefulness, drafts were reviewed by 

selected GAO, federal and state agency evaluators, and evaluation authors and

practitioners from consulting firms. 

46FMCSA headquarters officials stated that they conducted a preliminary analysis of the 

available data to determine potential sample sizes for specific intervention types and 

patterns. However, they did not provide documentation on the methods or results of this 

analysis. 

47UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 2011).
Page 30 GAO-17-49 Motor Carriers

in calendar years 2011 and 2012.48 By combining multiple years of 

intervention data, as FMCSA did in these evaluations, rather than using a 

fiscal year construction, which officials told us FMCSA used in the annual 

evaluation for administrative reasons, FMCSA could potentially overcome 

data sufficiency limitations (see app. III).

FMCSA and independent evaluators have identified a need for this level 

of detailed information in the management and implementation of 

interventions. For example, an internal FMCSA working group determined 

that the agency needed a more detailed understanding of the 

effectiveness of onsite focused investigations to empower investigators to 

select the most appropriate intervention to change carrier behavior.49

Additionally, in its 2014 assessment, the Independent Review Team that 

DOT tasked with reviewing FMCSA’s compliance review processes 

identified a need for FMCSA to perform consistent, detailed, evaluations 

of effectiveness by enforcement tool, such as intervention types. Without 

detailed evaluations, the team said that FMCSA would be unable to focus 

resources on using its most effective tools or to reconfigure tools that are 

not meeting the agency’s goals.50

Nonetheless, as discussed earlier in this report, FMCSA’s ability to 

accurately identify specific intervention types is limited. Accepted 

practices for designing evaluations state that quality evaluations should 

rely on credible data that are sufficiently free of errors that could lead to 

inaccurate conclusions. Taking steps to reliably and accurately identify 

each intervention type in the data used to support its evaluations would 

 48FMCSA, Analysis Brief: Effectiveness of Onsite Focused Investigations (March 2016); 

FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis (March 2016). 

Specifically, these evaluations measured the percentage of carriers that had BASIC 

percentiles at or above intervention thresholds 12, 18, and 24 months after receiving 

either intervention type, not accounting for external factors that may have influenced the 

BASIC percentiles. Based on these evaluations, FMCSA concluded that onsite focused 

and offsite investigations were an effective way to improve overall motor carrier safety 

performance for carriers that received them.

49FMCSA, Continuous Improvement Working Group Recommendations: Improving CSA 

Prioritization and Interventions (February 2015). For example, the working group stated 

that FMCSA needs to understand whether onsite focused investigations are more 

effective with certain BASICs or types of carriers. 

50Independent Review Team, Blueprint for Safety Leadership: Aligning Enforcement and 

Risk (Washington, D.C.: July 15, 2014).
Page 31 GAO-17-49 Motor Carriers

help FMCSA conduct evaluations that produce information appropriate to 

the design and implementation of CSA interventions.51

FMCSA’s annual and separate effectiveness evaluations had 

methodological limitations that limited their ability to accurately attribute 

changes in carrier safety behavior solely to interventions. Because of 

these limitations, FMCSA may not have accurately accounted for factors 

other than FMCSA’s interventions that could be responsible for the 

outcomes observed. Most notably, FMCSA did not consistently use a 

comparison group design, which compares outcomes among carriers that 

did and did not receive interventions, for its effectiveness evaluations. 

When FMCSA did use this design, it constructed comparison groups that 

did not sufficiently account for external factors that could affect group 

crash rates. According to recommended practices for designing program 

evaluations, comparison group designs are typical for assessing program 

effectiveness, because they can isolate a program’s unique effects when 

the comparison groups are sufficiently similar to the groups affected by a 

program. See appendix III for a complete assessment of the limitations 

we identified in FMCSA’s annual effectiveness evaluation, along with 

accepted practices that could help FMCSA to address them.52

FMCSA’s separate evaluations of onsite focused and offsite investigation 

effectiveness did not use a comparison group and so, as previously 

discussed, did not account for external factors that could have influenced 

changes in motor carriers’ BASIC percentiles. Officials stated that they 

determined that a comparison group method was not appropriate for 

FMCSA’s evaluation of onsite focused investigations because the officials 

were concerned that a comparison group would overstate the 

effectiveness of onsite focused investigations due to the differing safety 

profiles of carriers that receive each intervention type. FMCSA 

headquarters officials stated that they chose to use the same 

methodology as the onsite focused investigation effectiveness evaluation 

for the separate evaluation of offsite investigation interventions.

 51Taking these steps would be consistent with accepted practices for program evaluation 

to assess program effectiveness (see app. III). Because FMCSA’s intervention data were 

not sufficiently reliable to conduct our own effectiveness analysis, we could not determine 

how strongly these data limitations might affect the accuracy of FMCSA’s evaluations.

52We conducted a more detailed assessment of FMCSA’s annual effectiveness evaluation 

model because, according to FMCSA headquarters officials, it is the primary method 

FMCSA uses to evaluate CSA intervention effectiveness and FMCSA intends to use it on 

a recurring basis. 

Information Accuracy

Uses and Characteristics of Comparison 

Groups

A comparison group, in the context of typical 

designs for evaluating program effectiveness, 

represents what would have happened in the 

absence of a program and is used to rule out 

alternative explanations for changes in 

outcomes. In a truly randomized experiment, 

this w ould be the control group. In a quasiexperimental evaluation, like FMCSA’s annual 

effectiveness evaluation, w here participants

(i.e., carriers) are not sorted randomly into 

groups, the comparison group should be 

constructed to be as similar as possible to the 

group being influenced by the program (i.e., 

carriers receiving interventions), in order to 

draw strong conclusions about the effects of 

the program. The groups should be similar 

enough that any difference in outcome can be 

plausibly attributed to the intervention type 

being evaluated.

Source: GAO (GAO-12-208G; GAO analysis of FMCSA 

documents). | GAO-17-49
Page 32 GAO-17-49 Motor Carriers

Although FMCSA did use a comparison group approach in its annual 

effectiveness evaluation, we identified limitations with FMCSA’s approach 

that affect the model’s ability to accurately attribute changes in crash 

rates to interventions. For example, FMCSA’s comparison group was 

observed over a different measurement period from the carriers that 

received interventions, so that the two groups were not perfectly matched 

on broad changes, such as weather and economic changes. By not 

matching the measurement period, FMCSA’s use of a comparison group 

was limited in its ability to account for external factors, as intended.53

Additionally, in the 2015 evaluation, FMCSA’s comparison groups held 

constant the effects of carrier size, but did not hold constant key factors 

that could influence intervention outcomes, such as pre-intervention 

safety behaviors as measured by regulatory violations or BASIC 

percentiles. FMCSA headquarters officials said that comparison groups in 

the model accounted only for carrier size because of data limitations and 

their concern that accounting for too many additional variables would 

reduce the power of the model. Specifically, officials said that FMCSA 

does not currently have sufficient data to assess the effectiveness of 

agency interventions for motor carriers with different characteristics, such 

as the total number of miles a carrier’s vehicles travel per year. However, 

FMCSA has previously used the data it has available to hold constant 

important factors external to the program and attribute changes in 

outcomes to its interventions with greater accuracy (see app. III). For 

example, FMCSA’s August 2011 evaluation of the Operational Model 

Test, conducted by UMTRI, matched groups of carriers that did and did 

not receive various types of interventions on several key characteristics, 

such as the distributions of pre-intervention crash rates and BASIC 

percentiles. Without using a robust comparison group or a similar method, 

FMCSA cannot accurately determine whether changes in motor carrier 

safety performance are a result of interventions or whether other factors 

are responsible.

Without more complete, appropriate, and accurate information on the 

effectiveness of individual CSA intervention types, FMCSA lacks 

information it needs to make informed decisions and evaluate its 

 53Carriers that received an intervention were observed for a 12-month period following the 

date of intervention. However, because comparison group carriers did not have an 

intervention date from which to begin the “post-intervention” observation period, 

comparison group carriers were observed for an 18-month period. See appendix III for 

more information.
Page 33 GAO-17-49 Motor Carriers

performance in achieving its effectiveness outcome for CSA interventions. 

Additionally, FMCSA’s ability to accurately identify specific intervention 

types by analyzing MCMIS and EMIS data is limited due to the data 

limitations which we described earlier in this report. Without taking steps 

to address these limitations, FMCSA cannot accurately evaluate how 

effectively CSA interventions improve motor carriers’ safety performance.

As with its effectiveness outcome, FMCSA identified improved efficiency 

in its Strategic Plan: Fiscal Years 2015–2018 as one of its strategic 

outcomes for CSA interventions. We have previously reported that 

agencies should measure program efficiency by considering the 

relationship between two elements: (1) inputs, such as costs or hours 

worked, and (2) desired results, such as a program’s effect on conditions 

or behaviors.54 In the past, FMCSA evaluated the efficiency of CSA 

interventions using both of these elements. Specifically, UMTRI’s August 

2011 evaluation estimated the average cost of conducting individual 

intervention types and measured the effects on carrier safety 

performance of those CSA intervention types, as well as common 

intervention patterns, in four pilot test states during an 8-month period 

from October 2008 through May 2009.55 Since the UMTRI evaluation, 

FMCSA has continued to evaluate the effectiveness of CSA interventions. 

However, as we describe below, FMCSA has not taken similar steps to 

update its cost information—information FMCSA would need to 

understand the relationship between both efficiency elements.

The UMTRI evaluation developed average cost estimates for each 

intervention type by considering four cost variables: labor hours, 

government miles traveled, vouchers, and other expenses.56 The 

evaluation studied 920 interventions applied to 586 carriers, with the 

 54GAO, Streamlining Government: Opportunities Exist to Strengthen OMB’s Approach to 

Improving Efficiency, GAO-10-394 (Washington, D.C.: May 7, 2010).

55UMTRI, Evaluation of the CSA 2010 Operational Model Test (August 2011). The four 

pilot test states were Colorado, Georgia, Missouri, and New Jersey. According to the 

evaluation report, FMCSA did not estimate average costs for warning letters because they 

are automatically generated and thus have nominal associated cost.

56FMCSA’s evaluation did not clarify what specific costs vouchers and other expenses 

may include. To estimate labor hour costs, the evaluation used an adjusted hourly rate for 

staff in four labor categories. The adjusted rate reflected General Schedule basic hourly 

salary rates effective in January 2009 for each labor category, fringe benefits, and 

overhead. 

FMCSA’s Evaluation of 

Intervention Efficiency Is 

Based on Outdated and 

Non-Representative Cost 

Information
Page 34 GAO-17-49 Motor Carriers

number of carriers receiving each intervention type ranging from 6

carriers for notices of violation to 249 carriers for onsite focused 

investigations. The evaluation concluded that cooperative safety plans 

and notices of violation had the lowest average estimated costs—ranging 

from $95 to $118 respectively. Onsite comprehensive investigations and 

onsite focused investigations had the highest estimated costs, averaging 

$1,038 and $677 respectively.57

Since UMTRI conducted its August 2011 efficiency evaluation, FMCSA 

has continued to use the results to report and estimate the efficiency of 

interventions. For example, in its Budget Estimates: Fiscal Year 2017 

report, FMCSA requested $2.5 million and 50 additional Program 

Analysts to complete the last phase of the CSA program, including 

nationwide implementation of offsite investigations. To support this 

request, FMCSA stated that offsite investigations were “extremely 

efficient” and specifically cited the evaluation’s cost estimate.58 Similarly, 

FMCSA headquarters officials told us they currently use the UMTRI 

evaluation to estimate efficiency and to understand the relative costs of 

individual intervention types.

However, cost estimates from UMTRI’s August 2011 evaluation are no 

longer current, because the time and resources needed to conduct 

interventions has changed and the evaluation’s estimates are not 

representative of all states.

• The evaluation’s estimates are no longer current because the time 

and resources needed to conduct interventions has changed. For 

example, in April 2013, FMCSA implemented a substantial change to 

the way investigators conduct investigations, called EIT, which is 

intended to help identify the root cause of motor carriers’ safety 

 57Officials also noted that average cost estimates may not reflect the costs associated with 

applying interventions for specific types of carriers. For example, an onsite comprehensive 

investigation of a carrier that has one vehicle could take one day, while an onsite focused 

investigation of a large multi-vehicle carrier may take weeks. The UMTRI evaluation 

presented median, minimum, and maximum values to demonstrate the effect of outliers on 

the resulting estimates. For example, the evaluation estimated that the median ($192), 

minimum ($7), and maximum ($4,799) cost of issuing a notice of claim varied significantly 

from the average cost estimate ($428) for that intervention type.

58The budget request noted that offsite investigations also involved administrative work to 

coordinate the collection of necessary documents before the investigation began and 

stated that the additional Program Analysts could help with other programmatic activities, 

such as managing the high-risk carriers list.
Page 35 GAO-17-49 Motor Carriers

problems.59 FMCSA headquarters officials told us that, although using 

EIT takes additional time, it results in improved motor carrier safety 

performance.

• The evaluation’s estimates do not represent costs in all states. 

Specifically, the evaluation stated that its estimates pertain only to the 

four states studied. Thus the evaluation’s estimates may not 

appropriately represent the average costs associated with applying 

interventions in the remaining 46 states, the District of Columbia, and 

Puerto Rico.

Federal standards for internal control state that agencies should use 

quality information to achieve the agency’s intended program outcomes. 

Quality information includes information that is current.60 FMCSA 

headquarters officials told us that they have not taken steps to update the 

cost estimates from the UMTRI evaluation to determine current resources 

used in all states to conduct each intervention type because they believed 

that FMCSA policy and guidance were sufficiently well designed to enable 

division managers to select the least resource-intensive intervention type 

necessary to correct a carrier’s safety problem.

In March 2015, FMCSA’s then-Acting Administrator testified that given 

FMCSA’s limited resources relative to the size of the regulated motor 

carrier population, it is imperative for FMCSA to apply its resources 

efficiently.61 However, without cost estimates for CSA interventions that 

are current and representative of all states, FMCSA lacks information it 

needs to understand the most efficient methods of conducting CSA 

interventions in all states. Because FMCSA lacks current cost 

information, it also cannot evaluate or understand the relationship 

between these costs and the effectiveness of CSA interventions.

 59As previously mentioned, FMCSA implemented EIT as part of its continuous 

improvement efforts and in response to Independent Review Team recommendations. 

See Independent Review Team, Blueprint for Safety Leadership: Aligning Enforcement 

and Risk (Washington, D.C.: July 15, 2014).

60GAO, Standards for Internal Control in the Federal Government, GAO-14-704G 

(Washington, D.C.: September 2014).

61T.F. Scott Darling III, Acting Administrator, Federal Motor Carrier Safety Administration, 

Surface Transportation Reauthorization – Oversight and Reform of the Federal Motor 

Carrier Safety Administration, testimony before the Senate Committee on Commerce, 

Science, and Transportation, Subcommittee on Surface Transportation and Merchant 

Marine Infrastructure, Safety, and Security, 114th Cong., 1st sess., March 4, 2015.
Page 36 GAO-17-49 Motor Carriers

FMCSA has taken steps intended to improve the effectiveness and 

efficiency of CSA interventions—strategic outcomes—principally by 

establishing a working group to address these issues and implementing 

some of the group’s recommendations. However, FMCSA has not 

established performance measures to monitor progress toward achieving 

its intended efficiency outcome for interventions. Without establishing 

measures for both outcomes, FMCSA is and will be limited in its ability to 

monitor program performance and to balance these two priorities, as 

needed.

In April 2014, FMCSA formed a Continuous Improvement Working Group 

(CIWG) tasked with improving the effectiveness and efficiency of the CSA 

program, including CSA interventions. Comprised of FMCSA staff from 

divisions, service centers, and headquarters, as well as state partners, 

the CIWG’s objective was to assess the agency’s intervention and 

prioritization processes and to recommend improvements that increase 

program effectiveness and efficiency. To develop its recommendations, 

the working group reviewed available intervention data—such as 

investigation reports—and assessed current intervention practices by 

surveying and interviewing field staff, among other things. In February 

2015, the CIWG made 20 recommendations intended to achieve its 

effectiveness and efficiency objectives.62 According to FMCSA officials, 

the agency had implemented 12 of these recommendations as of April 

2016 and was working to implement the remaining 8 recommendations. 

Implemented recommendations include:

Changing FMCSA’s high-risk definition and prioritization criteria: In March 

2016, FMCSA adopted a change to its definition of high-risk carriers and 

the criteria it used to prioritize carriers for intervention.63 For example, as 

discussed above, for a carrier to be defined as high-risk under FMCSA’s 

new criteria, it has to exceed higher percentile thresholds than previously 

 62FMCSA, Continuous Improvement Working Group Recommendations: Improving CSA 

Prioritization and Interventions (February 2015). According to FMCSA, some CIWG 

recommendations corresponded with recommendations in the Independent Review 

Team’s July 2014 report, which stated that FMCSA should sharpen its priority-setting 

focus and improve the timeliness of investigator actions on motor carriers that represent 

the highest risk. The report also recommended that FMCSA establish regular reviews and 

feedback processes to ensure process consistency and quality. 

6381 Fed. Reg. 11875 (Mar. 7, 2016).

FMCSA Took Some 

Steps Intended to 

Improve CSA 

Intervention 

Outcomes, but Lacks 

Measures to Monitor 

Progress

FMCSA Has Taken Some 

Steps Intended to Improve 

the Effectiveness and 

Efficiency of CSA 

Interventions
Page 37 GAO-17-49 Motor Carriers

used in any of at least two of four specified BASICs. According to 

FMCSA, this decreases the total number of high-risk carriers but better

identifies carriers at a high-risk for crashes and will allow investigators to 

investigate higher-risk carriers sooner, using current resources. However, 

FMCSA’s continued reliance on BASIC percentiles supported by 

insufficient safety data, which we identified in our 2014 report and 

discussed above, will limit its ability to effectively identify and prioritize the 

highest-risk carriers for intervention.64

Changing criteria to receive warning letters: In January 2016, FMCSA 

expanded the criteria it used to determine which motor carriers receive 

warning letters in an effort to reach more carriers and, according to 

FMCSA, to prevent further non-compliance before a more intensive 

intervention type becomes necessary. Specifically, the CIWG 

recommended that FMCSA send warning letters to carriers with BASIC 

percentiles above the intervention threshold in more BASICs than 

previously allowed, while shortening the time period carriers have after 

receiving a warning letter to improve their safety performance before 

being prioritized for an investigation intervention. The CIWG projected 

that implementing the change would increase the number of warning 

letters that FMCSA issued by 30 percent; however, the CIWG noted that 

issuing a warning letter that is not effective may merely postpone the 

eventual necessity of a carrier receiving an investigation intervention and 

that issuing more frequent warning letters could dilute their effectiveness.

Establishing intervention quality review procedures: In March 2016, 

FMCSA issued two memorandums requiring field staff to use tools that 

FMCSA developed to evaluate and improve the quality of some 

intervention types. Specifically, FMCSA developed one tool to measure 

the extent to which investigators appropriately and accurately conducted 

onsite focused and comprehensive investigations and completed required 

documentation. Division managers are expected to use the tool to 

evaluate a selected sample of investigation reports quarterly, identify 

areas of needed improvement, and provide training to improve report 

consistency and quality. Similarly, FMCSA developed another tool that 

 64GAO-14-114. DOT’s Office of Inspector General is currently assessing whether 

FMCSA’s processes for ensuring that reviews of motor carriers flagged for investigation 

are timely and adequate. Specifically, the DOT’s Office of Inspector General is assessing 

whether (1) compliance investigations are conducted in a timely manner once a carrier is 

flagged; (2) comprehensive and focused investigations both ensure adequate review of 

flagged carriers; and (3) quality control measures used during the investigation and 

enforcement processes are adequate.
Page 38 GAO-17-49 Motor Carriers

measures the extent to which investigators’ enforcement cases for 

notices of claim and notices of violation include sufficient documentation 

to meet evidentiary requirements. The memorandum requires division 

managers to ensure that each notice is reviewed to identify areas for 

improvement and ensure that enforcement cases are properly completed. 

According to FMCSA officials, FMCSA intends to use the results of these 

evaluations to identify training or policy clarifications needed to 

continuously improve the application and effectiveness of each 

intervention performed.

As previously discussed, FMCSA’s Strategic Plan: Fiscal Years 2015–

2018 identified improved effectiveness and efficiency as strategic 

outcomes of CSA interventions. FMCSA headquarters officials told us 

that effectiveness and efficiency are complementary outcomes that 

FMCSA strives to balance. For example, according to officials, while 

using EIT requires more time and decreases the number of investigations 

that FMCSA can conduct (i.e., efficiency), it also increases investigation 

quality (i.e., effectiveness). Thus, senior FMCSA officials stressed the 

importance of considering both effectiveness and efficiency in any set of 

measures used to monitor interventions, and stated that without treating 

these two outcomes as parts of a whole, FMCSA cannot achieve its goals 

for CSA interventions.

FMCSA has established some measures for its effectiveness outcome, 

and monitors these measures on an annual and ongoing basis. While we 

identified several limitations with the design and methodology FMCSA 

used in its effectiveness evaluation above, FMCSA has established a 

measure for intervention effectiveness—crash rates—and annually 

monitors the agency’s performance against that measure. Officials told us 

that FMCSA also monitors effectiveness using investigation outcome 

measures, such as violation rates and safety ratings, on an ongoing 

basis.

However, FMCSA has not established measures to monitor progress 

toward achieving its efficiency outcome. According to headquarters 

officials, FMCSA considers the efficiency outcome to include two 

dimensions: (1) the number of carriers FMCSA reaches through 

interventions and (2) the resources required for FMCSA to conduct 

interventions, including the time and travel required to complete 

investigations. While officials stated that FMCSA monitors some 

information related to efficiency, such as the number of investigations 

completed and investigative outcomes, officials acknowledged that 

FMCSA Is Limited in Its 

Ability to Monitor Progress 

toward Achieving CSA 

Intervention Outcomes 

Because It Lacks 

Performance Measures
Page 39 GAO-17-49 Motor Carriers

FMCSA has not formally established measures for its efficiency 

outcome.65

Leading practices for performance management state that agencies 

should express outcomes in a measurable form and establish a set of 

performance measures that help monitor progress toward achieving each 

outcome.66 Additionally, our work has shown that agencies should create 

a set of performance measures that addresses important dimensions of 

program performance and balances competing priorities to increase the 

usefulness of performance plans in guiding decisions.67 While our past 

work has identified challenges some federal agencies face developing 

and using outcome-based efficiency measures, it has also highlighted the 

importance of developing such measures.68

Because FMCSA does not have a complete set of measures that reflects 

both its effectiveness and efficiency outcomes for CSA interventions, 

FMCSA managers lack benchmarks needed to regularly monitor progress 

toward achieving the outcomes. FMCSA also lacks information needed to 

balance these priorities and guide management decisions about 

FMCSA’s application of interventions. FMCSA’s limited resources and the 

increase in crashes involving motor carriers in recent years highlight the 

importance of ensuring that FMCSA regularly measures and monitors 

progress toward achieving both of these strategic outcomes.

 65FMCSA has established annual goals for the number of investigation interventions that 

each service center should aim to conduct, and headquarters officials told us FMCSA 

monitors progress toward achieving these goals by reviewing monthly performance 

reports. However, this measure does not directly reflect the number of carriers FMCSA 

reaches through interventions because carriers can receive multiple interventions and it 

includes only investigation interventions.

66GPRA Modernization Act of 2010 (GPRAMA), Pub. L. No. 111-352, §3, (codified at 31 

U.S.C. § 1115(b)(2), (6)). Although GPRAMA’s requirements apply at the departmental 

level (e.g., DOT) , we have previously stated that they can serve as leading practices at 

other organizational levels, such as component agencies, offices, programs, and projects. 

See, for example, GAO-12-77.

67GAO, Agency Performance Plans: Examples of Practices That Can Improve Usefulness 

to Decisionmakers, GAO/GGD/AIMD-99-69 (Washington, D.C.: Feb. 26, 1999). 

68GAO, Streamlining Government: Opportunities Exist to Strengthen OMB’s Approach to 

Improving Efficiency, GAO-10-394 (Washington, D.C.: May 7, 2010). 
Page 40 GAO-17-49 Motor Carriers

Fatalities involving motor carriers have increased—rising from about 

4,200 in 2011 to almost 4,500 in 2015—and interventions can play a 

critical role in reversing this troubling trend. FMCSA aims to reduce such 

fatalities by using a data-driven approach that identifies and intervenes 

with the highest-risk motor carriers early. To monitor its performance, 

FMCSA has identified improved effectiveness and efficiency as strategic 

outcomes for CSA interventions and has taken some steps to improve 

agency performance in these areas. For example, since March 2016, 

FMCSA has required field staff to use tools that FMCSA developed to 

evaluate and improve the quality of onsite investigations and two followon interventions.

However, we identified important limitations in the information that 

FMCSA used to evaluate the effectiveness and efficiency of interventions. 

For example, FMCSA’s effectiveness evaluations did not produce 

sufficiently complete, appropriate, and accurate information on individual 

intervention types or common intervention patterns, because of design 

and methodology limitations. Although FMCSA officials expressed 

concern about potential sample size limitations when evaluating the 

effectiveness of individual intervention types or common intervention 

patterns, the August 2011 UMTRI evaluation that FMCSA sponsored as 

part of its Operational Model Test demonstrated that such evaluations are 

feasible. Similarly, FMCSA uses cost estimates from UMTRI’s evaluation 

to understand the efficiency benefits of interventions. However, the 

evaluation’s cost estimates are no longer current because the time and 

resources needed to conduct interventions have changed and are not 

representative of costs nationwide. Without current cost estimates that 

are representative of nationwide costs, FMCSA lacks information it needs 

to understand the most efficient methods of conducting CSA interventions 

and cannot assess the relationship between these costs and intervention 

effectiveness.

Moreover, long-standing delays in developing SENTRI software have 

compromised the quality of intervention information, thereby limiting 

FMCSA’s ability to accurately and effectively monitor trends in its 

application of interventions over time and to evaluate intervention 

effectiveness. As FMCSA continues its efforts to address data limitations 

that affect the accuracy of intervention information, it is important that 

FMCSA not delay taking steps to improve how it currently evaluates the 

effectiveness and efficiency of CSA interventions by ensuring, for 

example, that its annual effectiveness evaluation addresses other 

limitations we have identified.

Conclusions
Page 41 GAO-17-49 Motor Carriers

FMCSA has dedicated significant resources to transition from a costly 

one-size-fits-all approach to a range of more effective and efficient 

interventions. However, without improving the quality of information that 

FMCSA uses to evaluate its performance, the agency will continue to lack 

the information it needs to determine the extent to which it is achieving 

these fundamental programmatic improvements.

In addition, we found that although FMCSA has established some 

performance measures for its effectiveness outcomes, the agency has 

not established measures to monitor progress toward achieving its 

efficiency outcome. FMCSA needs information on all dimensions of its 

effectiveness and efficiency outcomes to balance these priorities and 

guide management decisions about its application of interventions.

To determine whether CSA interventions influence motor carrier safety 

performance, the Secretary of Transportation should direct the FMCSA 

Administrator to:

• Identify and implement, as appropriate, methods to evaluate the 

effectiveness of individual intervention types or common intervention 

patterns to obtain more complete, appropriate, and accurate 

information on the effectiveness of interventions in improving motor 

carrier safety performance. In identifying and implementing 

appropriate methods, FMCSA should incorporate accepted practices 

for designing program effectiveness evaluations, including practices 

that would enable FMCSA to more confidently attribute changes in 

carriers’ safety behavior to CSA interventions.

To understand the efficiency of CSA interventions the Secretary of 

Transportation should direct the FMCSA Administrator to:

• Update FMCSA’s cost estimates to determine the resources currently 

used to conduct individual intervention types and ensure FMCSA has 

cost information that is representative of all states.

To enable FMCSA management to monitor the agency’s progress in 

achieving its effectiveness and efficiency outcomes for CSA interventions 

and balance priorities, the Secretary of Transportation should direct the 

FMCSA Administrator to:

Recommendations for 

Executive Action
Page 42 GAO-17-49 Motor Carriers

• Establish and use performance measures to regularly monitor 

progress toward both FMCSA’s effectiveness outcome and its 

efficiency outcome.

We provided a draft of this report to the DOT for review and comment. 

DOT provided written comments, which are reprinted in appendix IV. In its 

written comments, DOT concurred with our recommendations. DOT also 

described actions that FMCSA has taken to improve the CSA program 

and noted that CSA interventions have been shown to effectively improve 

motor carriers’ safety behavior. As stated in this report, the evaluations 

that FMCSA uses to assess intervention effectiveness did not produce 

sufficiently complete, appropriate, and accurate information on individual 

intervention types because of design and methodological limitations that 

limited FMCSA’s ability to accurately attribute changes in carriers’ safety 

behavior solely to interventions. We believe that identifying and 

implementing appropriate methods to address these limitations will help 

to provide FMCSA with information it needs to evaluate its performance in 

achieving its effectiveness outcome for CSA interventions. 

In addition, DOT provided technical comments on the draft report, which 

we incorporated as appropriate.

We are sending copies of this report to relevant congressional 

committees, the Secretary of Transportation, and the Administrator of 

FMCSA. In addition, the report is available at no charge on GAO’s 

website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-2834 or flemings@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made major contributions to 

this report are listed in appendix V.

Susan Fleming

Director, Physical Infrastructure Issues

Agency Comments
Page 43 GAO-17-49 Motor Carriers

List of Committees

The Honorable Susan Collins

Chairman

The Honorable Jack Reed

Ranking Member

Subcommittee on Transportation, Housing and Urban 

 Development, and Related Agencies

Committee on Appropriations

United States Senate

The Honorable Mario Diaz-Balart

Chairman

The Honorable David Price

Ranking Member

Subcommittee on Transportation, Housing and Urban 

 Development, and Related Agencies

Committee on Appropriations

House of Representatives
Appendix I: Objectives, Scope, and 

Methodology

Page 44 GAO-17-49 Motor Carriers

The objectives of our report were to examine: (1) the extent to which the 

Federal Motor Carrier Safety Administration (FMCSA) has implemented 

Compliance, Safety, Accountability (CSA) interventions and how it has 

applied them; (2) the extent to which FMCSA has evaluated the 

effectiveness and efficiency of CSA interventions; and (3) any steps that 

FMCSA has taken to improve and monitor progress toward achieving its 

intended outcomes for CSA interventions.

To determine the extent to which FMCSA has implemented CSA program 

interventions and how it applied them, we analyzed FMCSA intervention 

data from fiscal year 2010 through fiscal year 2015, the most recent fiscal 

year for which intervention information was available. Specifically, we 

analyzed data from two FMCSA data systems: (1) the Motor Carrier 

Management Information System (MCMIS) and (2) the Enforcement 

Management Information System (EMIS).1 If we determined that MCMIS 

and EMIS data were sufficiently reliable, we intended to then analyze 

whether there were any notable increases, decreases, or other trends in 

FMCSA’s application of interventions—across states, regions, and motor 

carrier types (e.g., fleet size). We also intended to determine common 

intervention patterns when carriers receive multiple interventions (e.g., 

warning letter, then off-site investigation, then notice of violation).

To determine the reliability of FMCSA data we requested a complete set 

of all MCMIS and EMIS data from fiscal year 2010 through fiscal year

2015. To develop our request, we conducted interviews with cognizant 

FMCSA officials as well as officials from the Volpe National 

Transportation Systems Center, which provides technical support to 

FMCSA’s data analysis. We would typically review documentation to 

prepare a data analysis plan; however, FMCSA could not provide up-todate or complete data dictionaries or other reference documents for these 

data systems. Thus, we requested and FMCSA provided sample data 

tables and variables that we could use to identify the occurrence of each 

intervention type by analyzing MCMIS and EMIS data.

 1

MCMIS is an information system that captures data from field offices. It is a source for 

FMCSA inspection, crash, compliance review, safety audit, and registration data. EMIS is 

a web-based application used to monitor, track, and store information related to FMCSA 

enforcement actions. It manages and tracks all data associated with notifying the carrier, 

monitoring the carrier's response, determining whether further compliance action is 

required, and generating reports for various headquarters, service center, and division 

staff. It is the authoritative source for FMCSA enforcement data.

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 45 GAO-17-49 Motor Carriers

Using the information that FMCSA provided, we performed electronic 

data testing to count how frequently FMCSA conducted each intervention 

type in fiscal year 2010 through fiscal year 2015. We then compared the 

results of our analysis against FMCSA-published sources to determine if 

the results included obvious errors or outliers. While we replicated 

FMCSA’s counts for warning letters and unsatisfactory/unfit out-of-service 

orders in all fiscal years, our comparison revealed differences in at least 

one fiscal year for all remaining intervention types. We subsequently met 

with FMCSA data analysis officials on several occasions to identify the 

cause of the differences that we identified. FMCSA officials told us that 

when they modified the algorithms used to count interventions, the 

agency applied the most recent algorithm to all previous data—including 

historical data—thereby changing the way that FMCSA counted 

interventions over time. As a result, FMCSA officials told us that we could 

not validate the results of our analysis against agency totals.

After evaluating the reliability of these data for our analytical and reporting 

purposes, we concluded that the data were of undetermined reliability, 

because data limitations prevented an adequate and comprehensive 

assessment. This precluded our analysis of trends in FMCSA’s 

application of interventions across states, regions, and motor carrier 

types. As a substitute, we requested that FMCSA provide estimates for 

how frequently it applied each intervention type from fiscal year 2010 

through fiscal year 2015 to identify general trends. After reviewing 

FMCSA documentation related to the estimates and interviewing 

responsible FMCSA officials, we determined that FMCSA’s estimates 

were sufficiently reliable for this purpose.

We reviewed relevant regulations and FMCSA guidance and policy 

documents to identify how FMCSA should implement and apply 

interventions. For example, we reviewed the electronic Field Operations 

Training Manual (eFOTM), which is the principal guidance document for 

assigning intervention types.

In addition, we interviewed FMCSA division officials in eight states that 

we selected based upon their participation in FMCSA’s Operational Model 

Test of the CSA program, geographic location, and program size, among 

other factors. Selected states included:

• Georgia

• Illinois

• Kansas

• Maryland
Appendix I: Objectives, Scope, and 

Methodology

Page 46 GAO-17-49 Motor Carriers

• Massachusetts

• Montana

• Oklahoma

• Texas

We selected these states to get a range of perspectives on FMCSA’s 

application of interventions. For example, four of the states participated in 

FMCSA’s Operational Model Test and thus had experience implementing 

all eight CSA intervention types. Similarly, we selected two states from 

each service center. Although the information obtained from our 

interviews with officials from the selected states is not generalizable to all 

states or FMCSA divisions, it provided illustrative examples of how 

FMCSA is applying interventions as well as the perspectives of officials 

knowledgeable about the program. In addition, we interviewed FMCSA 

officials from each service center—including FMCSA’s Eastern, 

Midwestern, Southern, and Western Service Centers—as well as 

headquarters officials from FMCSA’s Office of Enforcement, Office of 

Field Operations, and Office of Research and Information Technology. 

We also interviewed industry stakeholders, such as the Commercial 

Vehicle Safety Alliance, American Trucking Associations, Trucking 

Alliance, and the Owner-Operator Independent Drivers Association to 

gain their perspectives on FMCSA’s intervention and enforcement 

activities.

To determine the extent to which FMCSA has evaluated the effectiveness 

and efficiency of CSA interventions, we intended to conduct our own 

effectiveness evaluation to determine how interventions affect motor 

carrier safety and illustrate the strengths and limitations of particular 

evaluation designs. However, because MCMIS and EMIS data were of 

undetermined reliability, we instead reviewed the four evaluations the 

agency has conducted. Specifically, we reviewed:

• The University of Michigan Transportation Research Institute, 

Evaluation of the CSA 2010 Operational Model Test (August 2011);

• FMCSA, FMCSA Safety Program Effectiveness Measurement: Carrier 

Intervention Effectiveness Model, Version 1.0: Summary Report for 

Fiscal Years 2009, 2010, 2011 (January 2015);

• FMCSA, Analysis Brief: Effectiveness of Onsite Focused 

Investigations (March 2016); and

• FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis 

(March 2016)
Appendix I: Objectives, Scope, and 

Methodology

Page 47 GAO-17-49 Motor Carriers

We conducted a more detailed assessment of the second report on 

FMCSA’s annual effectiveness evaluation model because, according to 

FMCSA headquarters officials, it is the primary method FMCSA uses to 

evaluate intervention effectiveness and FMCSA intends to use it on a 

recurring basis. In addition, we reviewed FMCSA policy documents—

such as FMCSA’s Strategic Plan: Fiscal Years 2015–2018 and eFOTM 

guidance—to determine how FMCSA used the information produced by 

each of its four evaluations. We also interviewed FMCSA headquarters 

officials responsible for developing policy and conducting data analysis as 

well as officials from the Volpe National Transportation Systems Center 

responsible for designing and conducting some FMCSA evaluations.

To assess FMCSA’s effectiveness evaluations, we consulted GAO’s 

guidance on designing program evaluations, which describes accepted 

practices for evaluating program effectiveness based on GAO studies, 

policy documents, and program evaluation literature.2 We also consulted 

federal standards for internal control for using quality information, and 

drew on internal methodological expertise to assess the extent to which 

the designs, implementation, and results of FMCSA’s evaluations met 

quality information standards.3

To determine any steps that FMCSA has taken to improve and monitor 

progress toward achieving its intended outcomes for interventions, we 

reviewed relevant FMCSA strategic planning and policy documents that 

established such outcomes, principally FMCSA’s Strategic Plan: Fiscal 

Years 2015–2018. We reviewed reports from external entities that 

included recommendations intended to support improvements to 

FMCSA’s compliance and enforcement programs, such as the 

Independent Review Team’s July 2014 report and a March 2014 report 

from the Department of Transportation’s Office of Inspector General.4

 2

GAO, Designing Evaluations: 2012 revision, GAO-12-208G (Washington, D.C.: January 

2012). The report is based on GAO studies and policy documents and program evaluation 

literature. To ensure the guide’s competence and usefulness, drafts were reviewed by 

selected GAO, federal, and state agency evaluators, and evaluation authors and 

practitioners from consulting firms.

3

GAO, Standards for Internal Control in the Federal Government, GAO-14-704G 

(Washington, D.C.: September 2014).

4

Independent Review Team, Blueprint for Safety Leadership: Aligning Enforcement and 

Risk (Washington, D.C.: July 15, 2014) and Department of Transportation, Office of 

Inspector General Audit Report, Actions Are Needed to Strengthen FMCSA’s Compliance, 

Safety, Accountability Program, Report Number MH-2014-032 (Mar. 5, 2014).
Appendix I: Objectives, Scope, and 

Methodology

Page 48 GAO-17-49 Motor Carriers

Similarly, we reviewed FMCSA’s Continuous Improvement Working 

Group’s February 2015 report that included recommendations intended to 

achieve FMCSA’s effectiveness and efficiency outcomes.5

We also interviewed responsible division, service center, and 

headquarters officials to identify any steps FMCSA has taken to monitor 

or improve the effectiveness or efficiency of interventions as well as to 

determine their perspectives on the effects of these steps. For example, 

we interviewed responsible officials at headquarters who develop policy 

and oversee adherence to federal regulations by interstate motor carriers 

across the country to understand how they monitor or improve 

interventions. We also interviewed service center and division officials to 

understand the field’s involvement in FMCSA’s improvement activities. 

We compared the results of our documentary review and interviews 

against leading practices for performance management as well as key 

attributes of successful performance measures identified in our prior body 

of work.6 Although GPRAMA’s requirements apply at the departmental 

level (e.g., the Department of Transportation), we have previously stated 

they can serve as leading practices at other organizational levels, such as 

component agencies, offices, programs, and projects.

We conducted this performance audit from July 2015 to October 2016 in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on the audit objectives.

 5

See FMCSA, Continuous Improvement Working Group Recommendations: Improving 

CSA Prioritization and Interventions (February 2015). 

6

GPRA Modernization Act of 2010 (GPRAMA), Pub. L. No. 111-352, §3, 124 Stat. 3866, 

3867 (codified at 31 U.S.C. § 1115(b)(2), (6)). See also, GAO, Agency Performance 

Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers, 

GAO/GGD/AIMD-99-69 (Washington, D.C.: Feb. 26, 1999). 
Appendix II: Obligations to Develop Safety 

Enforcement Tracking and Investigation 

System Software

Page 49 GAO-17-49 Motor Carriers

Headquarters officials told us that the Federal Motor Carrier Safety 

Administration (FMCSA) obligated funds to develop Safety Enforcement 

Tracking and Investigation System (SENTRI) software each year from 

fiscal year 2009 through fiscal year 2013; however, they could not 

determine the exact amount of funds because FMCSA did not track 

information technology investments at the project level during those 

years. In addition, the agency obligated about $12 million from fiscal year 

2014 through fiscal year 2016 in contractor costs to develop the 

Compliance, Safety, Accountability (CSA) component of SENTRI (see 

table 5).1 FMCSA officials told us that FMCSA made some progress in 

developing the CSA component of SENTRI as a result of these 

investments. For example, FMCSA coordinated with field staff to identify 

system requirements.

Table 5: Federal Motor Carrier Safety Administration (FMCSA) Obligations to Develop Safety Enforcement Tracking and 

Investigation System (SENTRI) Software, Fiscal Years (FY) 2009–2016 (Dollars in Millions)

FY 2009 FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 FY 2015 FY 2016

Undetermineda Undetermineda Undetermineda Undetermineda Undetermineda $4.8 $4.3 $2.9

Source: FMCSA. | GAO-17-49

Note: Obligations may include non-development costs, such as operation and maintenance costs. 

Amounts are not adjusted for inflation.

a

FMCSA obligated funds to develop SENTRI software; however, FMCSA could not determine the 

specific obligation amount because it did not track project-level cost information at the time.

 1

According to FMCSA officials, FMCSA used a small amount of these obligations to 

manage and improve SENTRI’s New Entrant Program component.

Appendix II: Obligations to Develop Safety 

Enforcement Tracking and Investigation 

System Software
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 50 GAO-17-49 Motor Carriers

The Federal Motor Carrier Safety Administration (FMCSA), in conjunction 

with the John A. Volpe National Transportation Systems Center (Volpe) 

has modified an existing effectiveness model to develop a statistical 

model, the Carrier Intervention Effectiveness Model (CIEM), which 

annually measures the effectiveness of interventions.1 In January 2015, 

FMCSA published its first report using the CIEM, which evaluated the 

effectiveness of compliance reviews, and interventions FMCSA 

conducted in fiscal years 2009, 2010, and 2011. We analyzed the CIEM 

and the January 2015 evaluation report, using accepted practices for 

designing program evaluations and internal staff expertise.2 Below, we 

identify methodological strengths and limitations of these efforts, in 

addition to potential methods FMCSA could use to improve the 

capabilities of its model to estimate program impacts. Specifically, we 

have identified strengths and limitations in four key aspects of FMCSA’s 

effectiveness model: (1) general design, (2) comparison group, (3) 

observation periods, and (4) statistical analysis and inference.

According to FMCSA documentation, the CIEM is a statistical impact 

evaluation model that uses historical data to compare the safety 

improvement of carriers receiving FMCSA interventions (the treatment 

group) to carriers that do not (the comparison group). The January 2015 

evaluation report assessed two intervention types that existed prior to the 

Compliance, Safety, Accountability (CSA) program, including compliance 

review investigations and Performance and Registration Information 

Systems Management letters, and four new CSA intervention types, 

including warning letters, offsite investigations, onsite focused 

investigations, and onsite comprehensive investigations.

To estimate the impact of these interventions, the CIEM measures the 

difference between crash rates among carriers in the treatment group 

before and after receiving interventions, and then subtracts the difference 

in crash rates among carriers in the comparison group. The comparison 

 1

Because the January 2015 evaluation report assesses interventions FMCSA applied in 

fiscal years 2009, 2010, and 2011, spanning the transition to CSA, this report also 

includes some non-CSA interventions.

2

GAO, Designing Evaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: January 

2012). Designing evaluations is based on GAO studies and policy documents and 

program evaluation literature. To ensure the guide’s competence and usefulness, drafts 

were reviewed by selected GAO, federal and state agency evaluators, and evaluation 

authors and practitioners from consulting firms. 

Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

General Design
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 51 GAO-17-49 Motor Carriers

group accounts for confounding factors (other than FMCSA interventions) 

that may affect safety performance during the post-intervention period, 

such as broad changes in weather or economic conditions.

The model is designed to estimate the impact of interventions carried out 

in a single fiscal year and measures crash rates over a 12-month period 

following the first intervention a carrier receives in the fiscal year. The 

model estimates the combined impact of all interventions performed 

during 12-month periods, not the unique impact of each individual 

intervention type.

We identified the following limitations:

• Lack of process evaluation:The CIEM is designed to evaluate 

impact, but does not include a process study. According to accepted 

practices for designing program evaluations, a program logic model or 

process evaluation that identifies the most important external 

influences on desired program outcomes is valuable in planning an 

impact evaluation that convincingly rules out most plausible 

alternative explanations for the observed results.3 Process 

evaluations clarify the program as implemented and specify which of 

its activities may be responsible for the observed outcomes. The CSA 

program’s logic model might include elements of the CSA program, 

such as FMCSA and state safety inspectors and information systems, 

roadside inspections, safety interventions, and crashes, injuries, 

deaths, and monetary losses prevented.

Without an initial process evaluation, the impact evaluation cannot 

precisely identify what aspects of the program affect safety outcomes, 

or whether the estimated impacts reflect the program as designed. 

Without studying the program’s implementation and comparing its 

theoretical logic model to actual practices, it is uncertain whether 

impact estimates represent the effectiveness of the program’s 

activities as designed or the activities that program staff happened to 

have used in practice. The CIEM’s ability to accurately evaluate the 

impact of the CSA program could be improved by taking into account 

the program’s strategy and goals, and studying how the program is 

implemented.

 3

GAO-12-208G.
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 52 GAO-17-49 Motor Carriers

• Exclusion of intervention types: The January 2015 evaluation 

report did not include all intervention types. According to the report, 

the evaluation did not assess cooperative safety plans or direct 

notices of violation and direct notices of claim, because data on these 

intervention types had inconsistent completeness and accuracy. 

FMCSA officials told us that the evaluation included carriers that 

received follow-on notices of violation or claim, but it did not 

separately estimate their effects. When notices of violation or claim 

follow an investigation, the model implicitly estimates their effects in 

the post-intervention crash rate. However, the evaluation could have 

excluded some carriers that received a notice as their first intervention 

in the modeled year but not in the same fiscal year as the 

investigation.

• Aggregation of intervention types: The CIEM implicitly estimates 

how combinations of interventions affect safety, not the effects of 

each individual intervention type. The model identifies only the first 

intervention that a carrier receives in the modeled year, and estimates 

the intervention’s effect on crash rates from that time of first contact.

According to FMCSA officials, the model is designed to evaluate the 

effect of general FMCSA contact with carriers through interventions. 

Officials said that they tested some alternatives to using the first 

intervention in the fiscal year, including using a carrier’s most severe 

or last intervention in the fiscal year. Agency staff told us that they 

ultimately preferred to use the first intervention, because it 

represented the beginning of FMCSA’s influence on a carrier within 

the target time period.

FMCSA did not seek to estimate the effectiveness of individual 

intervention types, because agency staff had concerns about the 

small amount of data available on specific intervention types and 

common intervention patterns when carriers receive multiple 

interventions over time. FMCSA officials stated that Volpe conducted 

preliminary analysis of the data used for the CIEM to determine 

potential sample sizes, but did not provide documentation on the 

sample sizes for each intervention type or common patterns of 

interventions. The challenges arising from small sample sizes could 
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 53 GAO-17-49 Motor Carriers

potentially be addressed by pooling together data from multiple years, 

rather than relying on intervention data from one fiscal year.4

Recommended practices of program and policy evaluation 

recommend that program managers attempt to separately evaluate 

multiple types of program activities that seek to achieve a common 

outcome.5 Such “comparative effectiveness (or efficiency)” 

evaluations give managers information on how various activities 

perform compared to each other. Variation in outcomes across 

settings or populations can be the result of variation in program 

operations, such as varying types or levels of enforcement. Further, 

variation in outcomes associated with features under program control, 

such as different agency activities, may identify opportunities for 

managers to take action to improve performance.

To obtain this kind of information on comparative effectiveness, the 

treatment effects of interventions could be disaggregated into more 

nuanced categories than having at least one intervention. This 

approach could directly evaluate how each of several intervention 

types, or common intervention patterns, affect safety outcomes. 

Detailed performance information would better align with the design of 

the CSA program and give staff flexibility to choose from a range of 

intervention types that can address each carrier’s unique safety 

problems. With evidence of comparative effectiveness and efficiency, 

FMCSA would have better information on whether specific CSA 

interventions or combinations of interventions are more effective than 

others in certain circumstances.

The challenges from small sample sizes on particular interventions 

could be addressed in several ways. Pooling together data from 

multiple years, rather than relying on intervention data from one fiscal 

year, might produce a sufficient sample for evaluating less commonly 

used interventions. A multi-year design might become viable as the 

CSA program continues to produce data over several years, though 

pooling data might increase the potential for unmeasured factors to 

 4

See, for example, FMCSA, Analysis Brief: Effectiveness of Onsite Focused Investigations

(March 2016); and FMCSA, Effectiveness of Offsite Investigations: Preliminary Analysis 

(March 2016).

5

See, for example, GAO-12-208G, 38, 40 and Anna R. Solmeyer and Nicole Constance, 

“Unpacking the ‘Black Box’ of Social Programs and Policies: Introduction,” American 

Journal of Evaluation 36 (4): 470-474 and related articles in the edited journal volume.
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 54 GAO-17-49 Motor Carriers

influence safety outcomes. A process evaluation, as discussed above, 

could clarify how FMCSA field staff and state partners have 

implemented the program and could identify intervention types, or 

specific combinations of interventions, with sufficient data for analysis.

• Regression to the mean: The CIEM design does not fully account for 

the possibility that variation over time in the treatment group’s safety 

outcomes reflects regression to the mean. Regression to the mean is 

a statistical phenomenon in which, following an extreme measurement 

assumed to be due to random sources of variation, such as sampling 

error, subsequent measurements are likely to be closer to the 

average, or mean.

Under the CSA program, FMCSA prioritizes carriers to receive 

interventions based on whether the carriers’ percentiles exceed predetermined thresholds in any of seven behavioral analysis and safety 

improvement categories (BASIC).6 FMCSA’s decision to intervene 

with a carrier largely depends on BASIC percentiles, and officials use 

these percentiles, in addition to other criteria set out in guidance, to 

determine which type of intervention to apply. This is especially true 

for carriers that exceed the thresholds intended to identify the highest 

risk carriers, because FMCSA policy requires that those carriers 

receive onsite investigations.

However, some carriers, especially those with few inspections or 

vehicles, may go above the intervention threshold in one 

measurement period due to anomalous events, but return below the 

threshold in the next measurement period due to factors unrelated to 

intervention. For example, a carrier may generally violate vehicle 

maintenance regulations at the industry average rate over the longterm. The carrier’s estimated violation rate in FMCSA data—and its 

related BASIC percentile—may vary around the long-term average in 

any particular period. The degree of variation could reflect the actual 

inconsistency of the carrier’s maintenance practices over time or 

 6

Under CSA, carriers are selected to receive interventions, in large part, based on their 

BASIC percentiles. Using carrier violation data collected in roadside inspections or crash 

investigations, the safety measurement system (SMS) quantifies carrier performance by 

determining carriers’ violation rates in seven behavioral analysis and safety improvement 

categories, or BASICs, and then ranks carriers with other carriers that have similar 

exposure from lowest to highest rate. Carriers then receive a “percentile” based on the 

percentile in which a carrier’s violation rate falls in any given BASIC after being ranked 

against its peers. FMCSA has delineated percentile thresholds in each BASIC, at which 

point a carrier is prioritized to receive an intervention. 
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 55 GAO-17-49 Motor Carriers

sampling error in the estimation of its violation rate, related to its 

frequency of inspection (exposure to violating regulations). A large 

deviation in one period from the long-term average could exceed 

BASIC percentile thresholds and trigger additional FMCSA oversight, 

but the deviation may not reflect a real change in the carrier’s longterm maintenance behavior. In a subsequent period, the carrier’s 

BASIC percentile has a higher probability of returning to the long-term 

average than continuing at the extreme from the previous period, 

assuming the prior deviation was caused by random sources of 

variation, such as sampling error. The same process may apply to 

crash rates.

According to recommended practices for evaluating the impact of a 

program, the evaluation must be carefully designed to rule out 

plausible alternative explanations for the results.7 The CIEM’s quasiexperiment design implicitly controls for differences across carriers 

that do not vary substantially over short time periods, which could 

include carrier management practices, leadership, and operating 

routes and procedures. The design controls for industry-wide changes 

over time that are constant across carriers, changes that could include 

weather and economic conditions. Lastly, the design explicitly controls 

for carrier size by stratifying the analysis by size groups. Although the 

size control may account for differences within carriers over time 

among the treatment carriers, the model includes few other controls 

that might specifically address this potential threat to valid causal 

inference. By not fully accounting for regression to the mean, the 

CIEM could be attributing changes in outcomes to CSA interventions, 

when those changes would have occurred on their own without 

intervention.

Accordingly, there is some evidence that some carriers’ safety 

behavior improves without intervention. For example, the Independent 

Review Team found that, of the carriers FMCSA prioritized for 

intervention in 2013, nearly 33 percent had a BASIC percentile above 

threshold when assigned for intervention that was no longer above 

threshold at the time of the review, suggesting that carriers’ safety 

performance may improve naturally without intervention.8 As we have 

 7

Rossi and Freedman and GAO-12-208G, 39. 

8

Independent Review Team, Blueprint for Safety Leadership: Aligning Enforcement and 

Risk (Washington, D.C.: July 15, 2014)
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 56 GAO-17-49 Motor Carriers

previously reported, carriers with fewer vehicles experience wide 

variance in crash and violation rates, which can make them especially 

prone to regression to the mean.9

An alternative design might compare carriers just above and just 

below the intervention-triggering threshold (at a given point in time). 

This “regression discontinuity” design would lend itself better to 

interventions triggered automatically when carriers exceed some 

threshold and would better reflect the nature of the CSA program, as 

recommended in accepted practices for evaluation design.10

Another alternative approach that could address the regression to the 

mean issue would be to match carriers based on variation over time in 

the safety outcomes prior to exceeding a BASIC percentile threshold. 

This design would fall into a general class of methods that include 

pre-treatment outcomes as an additional covariate.11 This would 

enhance the comparison group’s control for any deviations in the 

outcome over time prior to treatment and thereby ensure that 

treatment carriers would be matched to comparison carriers with 

similar outcome dynamics.

The CIEM constructs comparison groups using carriers that did not 

receive any of the model’s interventions during the modeled, prior, or 

subsequent year. The model assigns carriers to one of several 

comparison groups, using the same measure of size—the number of 

vehicles—used to assign carriers to treatment groups. These separate 

comparison groups are intended to eliminate differences associated with 

carrier size from the model’s calculation of adjusted crash rates.

We identified the following limitation:

 9

GAO, Federal Motor Carrier Safety: Modifying the Compliance, Safety, Accountability 

Program Would Improve the Ability to Identify High Risk Carriers, GAO-14-114 

(Washington, D.C.: Feb. 3, 2014).

10GAO-12-208G, 43 and Richard J. Marcantonio and Thomas D. Cook, “Convincing 

Quasi-Experiments: The Interrupted Time-Series and Regression-Discontinuity Designs,” 

in Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer, eds., Handbook of 

Practical Program Evaluation (San Francisco: Jossey-Bass, 1994), 144-151.

11Elizabeth A. Stuart, “Matching Methods for Causal Inference: A Review and Look 

Forward,” Statistical Science, 25 (1): 6-7 and Guido W. Imbens, “Estimation of Average 

Treatment Effects under Exogeneity: A Review,” The Review of Economics and Statistics

86 (1): 5. 

Comparison Group
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 57 GAO-17-49 Motor Carriers

• Limited control: Accepted practices for program evaluation would 

classify the CIEM as a “quasi-experimental” design for estimating 

impact. Quasi-experimental designs compare outcomes in a treatment 

group to outcomes in a comparison group formed using non-random 

assignment. Due to the lack of randomized assignment, the treatment 

and comparison groups may differ on other factors that affect the 

outcome. To compensate for this potential bias, evaluators generally 

ensure that such confounding variables are held constant in the 

construction of the comparison group or use other methods of 

adjustment.12

The CIEM explicitly holds constant one factor in the construction of its 

comparison group: carrier size. The model implicitly controls for all 

factors that vary between the treatment and control groups but remain 

constant over time, such as state or region of operation. In addition, 

the model implicitly controls for all factors that vary over time and 

affect the treatment and comparison groups equivalently, such as 

industry-wide effects due to weather or economic conditions.

The CIEM’s “difference-in-difference” design provides these forms of 

implicit control, and thus is a key strength of the model. However, 

FMCSA might try to construct a more robust comparison group that 

explicitly controls for more than just carrier size. Since the design 

controls for variables that are fixed across carriers and vary in the 

same ways within groups over time, FMCSA might construct a 

comparison group that controls for change across multiple variables. If 

reliable data were available, then potential variables could include: 

multiple measures of size; state of registration, inspections, or 

violations; driver characteristics; pre-treatment safety outcomes; pretreatment BASIC percentiles; and pre-treatment inspections and 

regulatory violations. Controlling for pre-treatment outcomes and 

BASIC percentiles would be especially desirable and would address 

the potential limitation of regression to the mean because the 

fluctuations due to sampling error would be controlled by design.

Data availability and reliability may limit FMCSA’s ability to include 

additional control variables to construct comparison groups. For 

example, the 2015 evaluation report notes that data on vehicle miles 

traveled—a measure of carrier exposure to crashes—were less 

reliable than vehicle count data because, according to officials, they 

 12GAO-12-208G. 
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 58 GAO-17-49 Motor Carriers

were sometimes incomplete or inconsistent across carriers. Similarly, 

officials told us that FMCSA considered using carrier operation type to 

construct comparison groups, but ultimately did not because some 

carriers reported multiple operation types or changed their operation 

types over time. Given the central importance of the CSA program to 

FMCSA’s enforcement efforts, the agency would benefit from 

improving or expanding data collection to support a more robust 

model.

The CIEM defines different observation periods for the treatment and 

control groups. For the treatment group, the CIEM uses the date of the 

first intervention in the modeled fiscal year as the demarcation between a 

12-month pre-intervention period and a 12-month post-intervention 

period. Pre- and post-intervention crash rates are measured for these 12-

month periods. For the comparison group, in contrast, the CIEM defines 

18-month periods preceding and following the midpoint of the modeled 

fiscal year to measure pre- and post-intervention crash rates. (This is 

because comparison carriers do not have an intervention date to define 

the pre- and post-intervention periods and measure crash rates.) The 

evaluation report noted that these longer 18-month periods ensure that 

the comparison group’s crash rates cover the entire treatment group time 

frame. To adjust for the 50 percent longer observation period for carriers 

in the comparison group, the evaluation divided crash rates for those 

carriers by 1.5 to yield annual crash rates.

We identified the following limitation:

• Inconsistent time periods: According to accepted practices for 

program impact evaluation, a design must confidently rule out nonprogram influences that could cause changes in outcomes to occur.13

The CIEM does not completely account for factors that might have 

affected crash rates for both treatment and comparison groups, 

because the lack of overlap between the measurement time periods 

does not hold constant factors unique to the season or period of 

measurement. For example, as FMCSA officials noted to us, crash 

rates are known to depend on seasonal and periodic changes in 

weather, and the model’s lack of overlap in measurement time periods 

meant that treatment and comparison groups were subject to different 

seasonal and periodic conditions. Other potential confounding 

 13GAO-12-208G. 

Observation Periods
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 59 GAO-17-49 Motor Carriers

variables include seasonal or periodic variation in economic demand 

and state enforcement resources.

The design might use measurement periods that vary as a function of 

each observed intervention’s timing and characteristics. For example, 

the design might construct a custom comparison group for each 

member of the treatment group, selecting multiple comparison carriers 

using size and other potential confounding variables. A matching 

design in which each treatment carrier is compared to one or more 

control carriers would allow identical observation periods while still 

producing an impact estimate for interventions applied during a single 

fiscal year.

The CIEM’s use of statistical inference is appropriate to quantify the 

uncertainty of its impact estimates. Carrier behavior and safety outcomes 

are consistent with a partially random data generation process. In this 

context, statistical inference estimates the sampling variability of the data 

over multiple hypothetical realizations of the same process. Applying 

inferential statistical methods appropriately reflects the potential for the 

observational regulatory, intervention, and safety data to vary partially at 

random.

A critical estimate in the CIEM is the net crash rate change for the 

treatment group. The model defines this quantity as the difference 

between the treatment group’s pre- and post-intervention crash rates, 

after subtracting the crash rate change in the comparison group. The 

model then tests whether the net change differs from zero at the 0.05 

statistical significance level. The model excludes insignificant findings 

from its subsequent estimates of total safety benefits calculated.

The model estimates safety benefits by transforming the estimated net 

change in crash rates due to interventions into an estimate of total 

crashes avoided, using the treatment group’s pre-intervention crash rate 

per vehicle and post-intervention vehicle counts. The model uses 

historical crash severity data to further estimate injuries prevented and 

lives saved associated with each prevented crash.

The model extrapolates these safety benefits—crashes avoided, injuries 

prevented, and lives saved—to carriers that received interventions, but 

were excluded from the treatment group due to missing or outlier crash or 

vehicle count data, and, according to officials, intrastate carriers that were 

excluded from the treatment group in the January 2015 version of the 

Statistical Analysis and 

Inference
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 60 GAO-17-49 Motor Carriers

model.14 The January 2015 evaluation notes that FMCSA assumed that 

these carriers will exhibit the same response to intervention as the 

carriers included in the model. Accordingly, the model adds the estimated 

safety benefits for carriers included in the model to those for carriers with 

outliers and missing data. The sum determines the aggregate estimated 

safety benefits. According to the 2015 evaluation report, FMCSA 

extrapolated safety benefits to the following number of carriers that 

received an intervention but were not included in the model: 9,567 

carriers in fiscal year 2009; 9,929 carriers in fiscal year 2010, and; 14,816 

carriers in fiscal year 2011.

We identified the following limitations:

• Multiple hypothesis tests: The model uses multiple statistical 

hypothesis tests to calculate total safety benefits. The CIEM estimates 

impact on crash rates within four strata of treatment and comparison 

groups defined by carrier size. If the net change in crash rates within 

each stratum is statistically distinguishable from zero at the 0.05 

confidence level, the model uses the results to estimate total safety 

benefits by summing the estimated benefits across groups. 

Statistically insignificant results in any stratum provide zero safety 

benefits by assumption. In this sense, the model’s estimate of total 

safety benefit reflects the results of 4 separate hypothesis tests, each 

at the 0.05 confidence level.

When a statistical analysis, such as the CIEM, uses the results of 

multiple statistical hypothesis tests, the probability that any one test 

produces a false inference (i.e., a Type I error) cannot exceed the 

confidence level of each individual test.15 The CIEM has a 0.18 

probability of rejecting at least one null hypothesis of zero impact, 

even though each test uses a 0.05 confidence level.16 Accepted 

 14According to FMCSA documentation on updates to the CIEM model for fiscal years 

2010, 2011, and 2012, FMCSA added intrastate non-hazardous materials and foreigndomiciled carriers to the treatment and comparison groups. This change allows the model, 

to the extent that such carriers are different from others, to incorporate these differences 

rather than assume, via extrapolation, that they are identical to interstate and intrastate 

hazmat carriers in nature and their response to interventions. According to FMCSA this 

somewhat impacted the results of the model, but the 2012 CIEM report will update the 

historical results for fiscal years 2010 and 2011. 

15See, for example, William L. Hays, Statistics, 5th ed. (Fort Worth: Harcourt Brace, 1994), 

449-455. 

16The joint Type I error probability for K tests, each with an error probability α, is given by 

αg = 1 – (1 – α)

K

. When K = 4 and α = 0.05, αg = 0.18. See Hayes, 450. 
Appendix III: Analysis of the Federal Motor 

Carrier Safety Administration’s Carrier 

Intervention Effectiveness Model

Page 61 GAO-17-49 Motor Carriers

methods of hypothesis testing typically recommend adjusting the 

confidence level of each individual test, such as by applying 

Bonferroni adjustments, so that the group-wise error probability 

matches the analyst’s intended risk level for all planned tests. These 

adjustments typically produce lower alpha values for each individual 

test.

Multiple hypothesis testing methods would be appropriate for the 

CIEM, given that it ultimately seeks to estimate total safety benefits as 

a function of multiple hypothesis tests. Without making these 

adjustments, the CIEM’s estimates of total safety benefits may have 

more risk of error than FMCSA intends to accept because the model 

does not conduct a joint test. That is, the probability that at least one 

group’s safety benefits equals zero may exceed the 0.05 confidence 

level that FMCSA accepts in each individual test. An alternative 

approach might calculate the confidence interval of the summed 

safety benefit estimate and test whether it equals zero, consistent with 

the discussion below.

• Confidence intervals of estimated impacts: The CIEM uses 

inferential statistical methods to test the hypothesis that the impact for 

each pair of treatment and control groups equals zero. However, the 

CIEM does not conduct statistical inference, such as estimating 

confidence intervals, when analyzing the net change in crash rates 

and associated total safety benefits. This approach is inconsistent, 

given that the model’s hypothesis test for a non-zero net change in 

crash rates implies that the quantity is a random variable with a 

sampling distribution and confidence interval. The model’s authors 

agree with this implication, suggesting that the test is equivalent to a 

“95 percent confidence interval that does not include zero.” Total 

safety benefits must also have a confidence interval, given that it is a 

function of the change in net crash rates. Nevertheless, the CIEM 

reports only point estimates of safety benefits, without conveying the 

statistical uncertainty that the model’s assumptions imply.

Accordingly, the CIEM might estimate and report confidence intervals 

for its estimates of crash rate impact and safety benefits, in order to 

make the statistical inference consistent and quantify the uncertainty 

of its estimates. The current hypothesis testing approach may 

produce a point estimate for safety benefits that appears more precise 

than the underlying confidence interval would support.
Appendix IV: Comments from the Department 

of Transportation

Page 62 GAO-17-49 Motor Carriers

Appendix IV: Comments from the 

Department of Transportation
Appendix V: GAO Contact and Staff 

Acknowledgments

Page 63 GAO-17-49 Motor Carriers

Susan A. Fleming, (202) 512-2834 or flemings@gao.gov

In addition to the individual named above, H. Brandon Haller (Assistant 

Director), William Colwell (Analyst in Charge), Katherine Blair, Melissa 

Bodeau, Russell Burnett, David Hooper, Benjamin Licht, Grant Mallie, 

Ifunanya Nwokedi, Malika Rice, Sandra Sokol, Niti Tandon, and Jeff 

Tessin made key contributions to this report.

Appendix V: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(100211)
The Government Accountability Office, the audit, evaluation, and investigative 

arm of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to http://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and 

distribution and depends on the number of pages in the publication and whether 

the publication is printed in color or black and white. Pricing and ordering 

information is posted on GAO’s website, http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.