Motor Carrier Safety: More Assessment and Transparency Could Enhance Benefits of New Oversight Program
Released: 2011-09-29T00:00:00

Letter

Background

FMCSA and state law enforcement agencies in partnership enforce safety standards for the more than 500,000 interstate motor carriers operating in the United States. States and, to a lesser extent, FMCSA staff, perform roadside inspections of vehicles to check for driver and maintenance violations and then provide the data from those inspections to FMCSA for analysis and determinations about a carrier’s safety performance. FMCSA also obtains data from the reports filed by state and local law enforcement officers when investigating commercial motor vehicle accidents or regulatory violations. FMCSA provides grants to states that may be used to offset the costs of conducting roadside inspections and improve the quality of the crash data the states report to FMCSA. In addition, FMCSA’s field offices in each state, known as divisions, have investigators who conduct safety reviews of carriers identified by state inspection and other data as unsafe or at risk of being unsafe. Most states augment FMCSA investigators’ efforts by reviewing carrier operations as well.

Before CSA, FMCSA relied primarily on comprehensive compliance reviews on-site at carriers to determine whether they were operating safely. Carriers were selected for these reviews based on safety assessments generated by FMCSA’s statistical enforcement model— SafeStat—that used data obtained from accident reports and other safety data supplied by FMCSA’s state partners (see table 1). During these reviews, an investigator would visit a motor carrier to assess compliance with safety regulations by interviewing company officials and reviewing records that pertain to alcohol and drug testing of drivers, insurance coverage, crashes, driver qualifications, the number of hours a driver has worked within a certain time period, vehicle maintenance, prior inspections, and transportation of hazardous materials. FMCSA officials believe that such comprehensive compliance reviews are an effective way to assess a carrier’s safety performance. However, compliance reviews are extremely resource intensive; therefore, only a small percentage of the motor carrier industry can be evaluated in this manner, given limited federal and state resources. Annually, for example, FMCSA and its state partners have conducted compliance reviews of about 3 percent of registered motor carriers. As a result, FMCSA was not able to evaluate the vast majority of registered motor carriers and most were not assigned a safety rating.

In 2004, FMCSA began to design and develop CSA, a program to better target resources toward unsafe carriers, deploy a more comprehensive array of interventions, and proactively evaluate safety performance based on data, rather than solely based on compliance reviews. Through implementation of CSA, FMCSA expects to assess a larger portion of the motor carrier industry and to increase the emphasis on driver safety. Additionally, FMCSA expects to use data to identify unsafe carriers and drivers earlier to address safety problems before crashes occur. In this way, FMCSA intends to create a culture of compliance, in which officials and carriers will work together to address safety issues early, and carriers will have access to information and resources that can help them better comply with safety regulations. FMCSA officials expect this approach will more efficiently use FMCSA and its state partners’ resources. FMCSA expects to significantly reach, or “touch,” more carriers—thus improving their safety—and ultimately reduce motor carrier crashes, injuries, and fatalities.

To date, FMCSA has focused its implementation efforts on carriers— examining the safety performance of the company—whether it be a trucking company with hundreds of vehicles or a small company operating one or two trucks. FMCSA’s implementation efforts also include an increased assessment of the safety behavior of the drivers for carriers selected for intervention. FMCSA also intends to rate or determine the fitness of all drivers, regardless of whether the carriers they work for are selected for intervention. The rating would cover such things as whether the driver was driving while impaired by drugs or alcohol or received tickets for moving vehicle violations.

SMS—the first oversight activity under CSA—is intended to allow FMCSA to more accurately assess a carrier’s safety performance. SMS is applied to safety data obtained primarily from roadside inspections as well as from crash reports. These data are sorted into six Behavior Analysis and Safety Improvement Categories (BASIC) that are associated with unsafe performance according to FMCSA’s analysis. In addition to the six BASICs, SMS also incorporates data based on a carrier’s crash involvement (see table 1).

Once the data are sorted into the seven data categories, the SMS algorithm measures and generates scores for the carrier’s safety performance in each category. Carriers are placed into peer groups (i.e., other carriers with similar numbers of inspections or size) and ranked according to performance. The rankings determine which carriers are not operating with optimal safety practices and, therefore, will be prioritized for intervention. CSA is intended to improve upon SafeStat, which measured safety in only four safety evaluation areas: driver, vehicle, safety management, and accident (equivalent to the SMS Crash Indicator). CSA uses a wider array of safety data to create a more nuanced understanding of a carrier’s safety performance and presents that information using more refined categories.

FMCSA has made carriers’ SMS scores available to carriers themselves as well as to the public, including shippers and insurers. Carriers are allowed to request reviews of any data they believe are incorrect through an FMCSA system known as DataQs. These requests for review can include moving violations reported by state authorities that carriers believe are invalid or mistakenly attributed to the wrong carrier. FMCSA forwards each request for review to the state in which the carrier was cited. States then research the issue, often by contacting the inspector who conducted the inspection and his or her supervisor. Based on this research, states decide if the violation is warranted and make changes if necessary.

All of these safety data are collected and maintained in FMCSA’s existing Motor Carrier Management Information System (MCMIS). Our previous work assessed FMCSA data reliability and identified problems with the quality of crash data reported to FMCSA, including data that were inaccurate, incomplete, and not reported in a timely manner. FMCSA has been making efforts to improve crash data quality, including awarding Safety Data Improvement Program grants to states to improve their crash data. States’ efforts to improve crash data include expanding electronic reporting; improving the timeliness, completeness, and accuracy of reporting; and standardizing police accident report forms.

The second oversight activity under CSA is the introduction of a variety of interventions for interceding with carriers when their SMS scores indicate safety deficiencies. The expanded array of interventions available under CSA offers FMCSA more flexibility and the opportunity to apply interventions commensurate with a carrier’s safety performance (see table 2). The new interventions were created to get carriers to improve behaviors linked to possible crash risk. As a result, these carriers have the opportunity to take corrective actions to avoid another intervention in the future. Under CSA, interventions that involve investigations follow a process known as the Safety Management Cycle which will expand investigations from simply identifying what violations occurred to determining why violations exist so that FMCSA can offer more constructive improvement recommendations.

While some of the interventions, such as Notice of Violation and Notice of Claim, available under CSA are not new, FMCSA intends to apply them in a more systematic manner under CSA. For example, according to FMCSA, the agency only issued a handful of Notices of Violation over the past 5 years because prior FMCSA information technology systems did not provide the capacity to issue and track them. Under CSA, Notices of Violations can be issued in conjunction with Cooperative Safety Plans, giving carriers a framework in which to address the violations. In another example, the agency intends to increase its use of the Notice of Claim.

The third oversight activity under CSA is determining a carrier’s fitness to operate motor vehicles, known as a Safety Fitness Determination. FMCSA plans to use SMS scores to make a Safety Fitness Determination to indicate whether a carrier should continue to operate or should be suspended from operating (i.e., be ordered “out-of-service”). Currently, FMCSA determines a carrier’s fitness to operate based only on the outcome of an onsite comprehensive investigation, similar to how it was done under SafeStat. If a review shows that a motor carrier is unfit to operate pursuant to governing regulations, FMCSA can issue an Out-of- Service order that prohibits the carrier from operating until the deficiencies are corrected. However, as part of CSA, FMCSA plans to initiate a rulemaking that will enable it to use SMS-generated scores to determine if carriers are unfit to operate. FMCSA has not determined if the same categories currently used to determine if a carrier is fit to operate—“satisfactory,” “conditional,” and “unsatisfactory”—will be used, but it does not plan to increase the number of categories.

In 2008, FMCSA launched an operational-model test (pilot) of the CSA program in four states and later expanded the pilot to five more states over 30 months through June 2010. During Phase 1, four states (Colorado, Georgia, Missouri, and New Jersey) tested CSA on carriers with the exception of those with the poorest SafeStat ratings. Fifty percent of the non-excluded carriers in each state were subject to certain aspects of the CSA model—specifically a subset of the BASICs and the interventions— and the other 50 percent were subject to SafeStat. During Phase 2, the carriers subject to CSA in those four states, including those excluded from Phase 1, were then subjected to all of the BASICs and interventions. Later, FMCSA added Delaware, Kansas, Maryland, Minnesota, and Montana to the pilot testing, with 100 percent of the carriers in each state subject to all of the BASICs and interventions. UMTRI analyzed the results of Phase 1 of the pilot as well as supplementary results from Phase 2 and issued its final report in August 2011.

In February 2011, we reported that FMCSA obligated more than $30 million for costs related to CSA from fiscal years 2007 through 2010. FMCSA used these funds to develop the SMS and new interventions, conduct and evaluate the pilot test, conduct travel and training related to CSA, and develop information technology related to CSA.

Some CSA Oversight Activities Are Functional but Others Are Indefinitely Delayed, and Implementation Problems Could Compromise the Program’s Effectiveness

Close to a year after the anticipated completion date, FMCSA has partially implemented two of the three planned CSA oversight activities—the SMS and an expanded set of interventions—in all states; however, it still cannot use CSA safety ratings to (1) use CSA to assess the fitness of motor carriers or (2) assign safety fitness determinations to individual drivers that would prohibit them from operating trucks and buses. Although it has been delayed, FMCSA has begun to implement the CSA oversight activities directed at carrier safety, including SMS and carrier interventions, such as Warning Letters and On-site Focused investigations. However, FMCSA has yet to issue the Notice of Proposed Rulemaking (NPRM), originally scheduled to be finalized in 2009, that would allow it to use CSA data to get unsafe carriers off the road. At present, it appears that FMCSA will not be issuing the rulemaking until later this year at the earliest. Furthermore, in implementing these CSA oversight activities, FMCSA has experienced issues that could affect CSA’s effectiveness. However, FMCSA has not provided comprehensive information to Congress and the public on the status of CSA as well as the risks associated with these delays and issues, and how it plans to mitigate those risks. Moreover, FMCSA has only recently taken steps to separately measure the fitness of drivers to operate trucks and buses, as research has shown that drivers—not vehicle problems—cause most carrier crashes. FMCSA has not specified time frames for developing this component or how it will ultimately be used.

FMCSA Has Fully or Partially Implemented Some Carrier Oversight Activities but Operational Issues Could Compromise Effectiveness

Although two of CSA’s three planned oversight activities for evaluating carriers are at least partly implemented and functional to varying degrees, implementation remains a work in progress. The first CSA oversight activity—developing SMS—was implemented in December 2010, as scheduled, and is functional (see table 3). For the second oversight activity, seven of the nine interventions—five of which are new—are generally functioning as intended. Two others—Off-site Investigations and Cooperative Safety Plans—have been delayed indefinitely because the technology needed to implement them is not yet operational. With respect to the third planned oversight activity, suspending unfit carriers on the basis of SMS scores, FMCSA originally intended to finalize the rulemaking by 2009 but this effort has been delayed; FMCSA now plans to issue the Notice of Proposed Rulemaking later this year and will not finalize the rulemaking until 2013. According to FMCSA officials, they delayed the rulemaking because of needed changes to SMS that arose during the pilot. In addition, they indicated that FMCSA has a backlog of other key rulemakings that has affected its ability to complete the CSA rulemaking.

FMCSA fully implemented the system to measure the performance of carriers in all safety categories in 2010. This information is provided to carriers to help them identify and address their own safety issues. Additionally, FMCSA has made most carriers’ safety data publicly available since December 2010 (see fig. 1 for a sample screenshot of carrier information available to the public). Shippers and insurers, among others, can now use this information to make business decisions. However, as figure 1 shows, the Crash Indicator score and the Cargo- Related BASIC score are not being made publicly available. Stakeholders raised concerns that the Crash Indicator includes all crashes, including those in which the driver was not accountable. FMCSA took an interim step to make the Crash Indicator score available only to the carrier. FMCSA plans to contract with the Department of Transportation’s John A. Volpe National Transportation Systems Center (Volpe) to develop a system to allow states to determine if a driver is accountable for a particular crash. FMCSA expects Volpe to begin work on this effort in January 2012. Specifically, FMCSA intends to allow carriers to request changes to their violations data by providing a police accident report to demonstrate that the carrier should not be held accountable for a particular crash. Similarly, the motor carrier industry raised concerns about biases created by grouping different types of carriers together for the Cargo-Related BASIC, specifically grouping open deck carriers (flat bed carriers) with those that use enclosed trailers. FMCSA agreed with the industry that these biases may exist and decided not to make the Cargo-Related BASIC data publicly available. In addition, industry raised concerns about FMCSA’s original plans to base individual carrier crash rates on the number of power units, i.e., trucks they operate, as opposed to the number of vehicle miles traveled. FMCSA agreed that vehicle miles traveled is a more equitable measure of exposure when determining crash rates. After considering industry concerns, FMCSA modified the measurement system to now use a combination of power units and vehicle miles traveled to analyze crash risk. According to most trucking association officials we interviewed, FMCSA has been willing to listen to carriers’ concerns while implementing CSA and, according to several, has responded by making adjustments.

Another issue that has arisen during the implementation of this part of CSA is that state enforcement agencies, such as state police or state highway patrol agencies, have experienced some difficulties handling motor carriers’ requests to review violations data through FMCSA’s DataQs system. In the months before FMCSA began implementing CSA nationwide, as well as after FMCSA began implementing CSA, carriers have been requesting reviews of violations data at a higher rate than in the past and, in some cases, straining states’ resources. Although carriers previously could request reviews of violations data through the DataQs system, carriers did not challenge the data as often because SafeStat focused on only certain violations. Because CSA uses all violations to determine carriers’ SMS scores and has made an expanded range of data about the motor carriers’ safety records available to them, carriers have taken a much greater interest in these data. Specifically, in August 2010, when FMCSA first made the violations data available for carriers’ review, the number of requests for review was about 2,600 per month. This number increased to a high of about 5,000 per month in October 2010, 2 months after FMCSA made carriers’ BASICs scores available for their review. Although this number has since decreased to about 3,700 per month by May 2011 and decreased further to about 3,000 by August 2011, it is still higher than when FMCSA first made violations data available for carriers’ review. Specifically, state officials in four of the eight states we visited told us they have experienced significant increases in the volume of these requests, which has strained their resources. For example, in Maryland, the volume of requests for data review has increased from 65 in August 2010 to 122 in May 2011 before decreasing to 78 by August 2011. To deal with the increase, the Maryland State Police added another person to handle the requests. Similarly, in Texas, the number of requests for data review increased from 195 requests in August 2010 to 285 by May 2011 before decreasing to 225 by August 2011. To handle the increase, Texas officials reassigned staff to handle the increased workload but planned to wait before hiring someone permanently. In addition to the impact on state resources, state officials in California said the increase in requests could affect their ability to resolve them within 10 days, FMCSA’s goal for responding to carriers. Although the volume of data review requests from carriers has been declining, it is unclear if this trend will continue as implementation of CSA progresses.

Trucking associations have raised concerns about how states handle these requests, as well as about states’ willingness to change violations data. According to state law enforcement officials, states review the requests and correct violations that are in error. Officials also indicated that some requests reflect carriers’ efforts to have as many violations removed from carriers’ records as possible. In January 2011 FMCSA—in conjunction with its State Partners—developed and issued a guide to address issues concerning consistency among states in handling requests to review violations data.

Thus far, FMCSA has fully implemented seven of the nine interventions nationwide. Of these seven, three are new—the Warning Letter, Targeted Roadside Inspection, and Onsite Focused Investigation. The Notice of Violation, Notice of Claim, Onsite Comprehensive Investigation, and Operations Out-of-Service Order existed before CSA and thus were already implemented nationwide. Together, as table 4 shows, these interventions provide a range of benefits. While FMCSA previously expected to implement two other new interventions—Off-site Investigations and Cooperative Safety Plans—nationwide by August 2011, it has delayed their implementation in the nonpilot states because it has not yet finished developing the key technology required to manage them. This technology, known as Sentri, is part of FMCSA’s ongoing information technology modernization effort and is intended to provide FMCSA enforcement and field staff easier access to carrier and driver information and to help FMCSA and states target unsafe carriers and drivers. FMCSA officials indicated that, although the agency’s current legacy systems contain the information investigators need to conduct Off- site Investigations and Cooperative Safety Plans, the systems do not interact very well. According to FMCSA, one of Sentri’s benefits is that it will create an environment with a single interface where users can conduct inquiries, inspections, investigations, and interventions, and create and review reports. Additionally, Sentri will align information technology systems with the changes to the investigative processes resulting from the interventions. FMCSA expects to complete this technology in April 2012. FMCSA officials indicated that the delays were due to communication problems between information technology and program offices—who are customers—as to the data requirements for the system. Specifically, officials said that program offices needed to better explain and define requirements so that everyone understands them. According to FMCSA, its information technology office has put in place new collaboration and communications methods with the sponsoring program units. We have reported in the past on the importance of establishing an agreed-upon set of requirements for customers and stakeholders. Until FMCSA completes this technology and can fully implement all of the interventions, it will not be able to reach the increased number of carriers originally intended.

One issue that could influence the effectiveness of the interventions is training. As a result of the delay in completing Sentri and the decision to delay implementing off-site investigations and cooperative safety plans, FMCSA revised its training plans for nonpilot states. Originally, FMCSA planned to provide 1 week of classroom training to FMCSA division and state officials and staff in nonpilot states, as it had done in the pilot states. Instead, when FMCSA decided to roll out the interventions in a phased approach, FMCSA division management received 1 day of classroom training, while other FMCSA division and state investigators received a series of webinars on the first phase of the roll out. Additionally, FMCSA and state officials in pilot states are serving as mentors to assist their counterparts in nonpilot states.

FMCSA and state officials we interviewed in nonpilot states had mixed opinions on the training. Six FMCSA and state officials in two of the nonpilot states we visited indicated that, because only certain interventions were implemented and pilot states were providing assistance, they felt the training prepared them to implement the interventions FMCSA initially rolled out. For example, officials in one state believed that, because the On-site Focused Investigations and Comprehensive Investigations were similar to the compliance reviews conducted in the past, they were comfortable with the training they have received. However, two of the FMCSA officials and one state official in the nonpilot states we visited felt the training lacked detail and was insufficient because CSA was still evolving. For example, officials in one state noted they were not yet conducting On-site Focused Investigations because they did not feel comfortable with the training they had received on this intervention. FMCSA officials indicated they were not aware of any other states that were not conducting On-site Focused Investigations. However, officials in two states said that while investigators were conducting On-site Focused Investigations, they were concerned about how effectively they were being conducted given limited training or because investigators were not yet comfortable with conducting focused reviews instead of comprehensive reviews.

FMCSA is taking steps to improve training on interventions. FMCSA officials acknowledged that the training to date was insufficient and explained that when they decided to begin implementing CSA in the fall of 2010, they used the webinar approach to provide information quickly to FMCSA divisions and states. FMCSA provided 2 days of additional training during the summer of 2011 that consisted of classroom training in all 50 states and included both management and investigators in FMCSA divisions and state agencies. This training includes the Safety Management Cycle approach to interventions involving investigations which, as noted, FMCSA believes will allow investigators to determine why violations occur and offer recommendations for improvement. FMCSA expects that the Safety Management Cycle will be implemented by the end of 2011. FMCSA officials also indicated that, as they developed this training, they incorporated the suggestions from participant evaluations from earlier training classes and agency surveys from both pilot and nonpilot states.

FMCSA is roughly 2 years behind its original target date for issuing and completing the rulemaking required to use SMS to determine a carrier’s fitness to operate. We reported in December 2007 that FMCSA planned to publish a NPRM for the carrier safety fitness determination in summer 2008 and expected the final rule to be in place in 2009. However, because of changes to SMS that arose during testing—such as the change in calculating crash rates—and a backlog of rulemakings for other FMCSA programs, officials now plan to issue the NPRM late in 2011 and finalize the rule in 2013. However, the date FMCSA can finalize the rule could also be delayed. FMCSA officials indicated they do not foresee any major challenges in meeting the current schedule because they have held public information sessions since 2008 to inform the motor carrier industry of the methodology they are considering for the safety fitness determination. On the other hand, others, such as the National Transportation Safety Board and the National Private Truck Council, noted that rulemakings could take much longer. Until the rulemaking is completed, FMCSA will not realize one of its most important goals for CSA—enhancing its ability to assign safety fitness determinations to a significantly greater portion of the motor carrier industry than it currently is able to do.

FMCSA Has Had Mixed Success Managing CSA’s Implementation and Has Not Communicated to Congress Its Strategy to Address Identified Weaknesses and Delays

In some areas, FMCSA performed well as it implemented CSA, most notably, in conducting extensive outreach to carriers. In December 2007, we reported that communicating needed information to key stakeholders would be critical to implementing a successful program. According to trucking association representatives, FMCSA has made considerable effort to provide information to carriers and associations and, according to one state trucking association, has probably done as much outreach as possible, given its resources. FMCSA’s efforts to reach out to carriers and make them aware of the program, if continued, could help FMCSA educate carriers about future developments in the program and forestall problems as it completes implementing the carrier component of CSA nationwide. Our interviews with 55 carriers indicated that 23 had learned about CSA from a variety of sources, including FMCSA’s and states’ outreach efforts and state trucking associations. However, 32 of the carriers indicated that they were not familiar with CSA. Of these carriers that had never heard of CSA, 12 were small carriers, 15 were medium and 5 were large. While the results of our interviews are not generalizable, they suggest that FMCSA should continue its outreach efforts.

FMCSA has also been responsive to stakeholder concerns during CSA’s implementation. In our December 2007 report, we said that controlling the project by monitoring and providing feedback would be critical to CSA’s success. Throughout the pilot and implementation, FMCSA has made changes to CSA based on feedback from carriers and states. As noted previously, in addition to deciding to not make the Crash Indicator and Cargo-Related BASIC data public, FMCSA also expanded its basis for calculating crash rates to include both power units (i.e., trucks) as well as vehicle miles traveled after stakeholders raised concerns. After studying the issue, FMCSA determined that including vehicle miles traveled in addition to power units was a more accurate measure.

Although FMCSA has managed CSA implementation well in these areas, the agency has experienced some difficulties in others.

FMCSA conducted a workforce analysis study in 2009 to determine the staffing levels and skill sets necessary to implement CSA. Based on this study, FMCSA planned to hire additional staff, including staff to support the expected increase in investigations. For fiscal year 2012, FMCSA has requested $78 million from Congress to fully implement and integrate CSA into its operations. Of this request, $61 million is for 696 full time positions, including salary and benefits, which represents most of FMCSA’s existing field staff as well as 98 new full-time positions. These new positions include 30 investigators and 51 program analysts who would assist intervention managers and investigators throughout FMCSA’s divisions, among other staff.

Notwithstanding the future of its funding request, FMCSA has not yet fully determined how it would allocate staff as it moves forward to implement CSA. FMCSA has not determined which divisions will receive the additional investigators and program analysts, although small states will likely share program analysts. FMCSA also has not performed a staffing analysis to determine how it would reallocate existing staff if it does not receive the funding in fiscal year 2012 for the new positions. We have identified key practices for workforce planning, including developing a process to determine staffing needs and allocate staff among offices and taking the budgetary process into account. Given the current budgetary environment, FMCSA officials realize they may not receive all funding requested and plan to re-examine current staff allocations if FMCSA does not receive authority for these positions. FMCSA officials have stated that CSA’s effectiveness would be impacted with less funding because investigators would not be able to conduct the same number of interventions and, consequently, FMCSA would not be able to reach as many carriers as originally expected. However, waiting to determine how to allocate a lesser number of staff could also delay FMCSA efforts to continue to implement CSA.

In addition, FMCSA is still adapting to the changes required by the new interventions. CSA represents a shift to a new paradigm or way of thinking about safety that requires a cultural change among FMCSA Division and state staff, which can take time. CSA requires investigators to change from comprehensively investigating all aspects of a motor carrier’s operations to focusing only on weaknesses that SMS identifies (i.e., the on-site focused investigation). During our site visits, FMCSA division and state staff often reported that they appreciated the efficiencies gained by using data to identify carriers and areas to focus on during investigations. However, they also reported that this shift has been difficult, with some investigators still preferring to conduct comprehensive investigations. FMCSA officials noted that investigators can expand a focused review if they see evidence of problems in other areas and that the efficiency gains FMCSA intends will be negated if investigators continue to take a comprehensive approach when focused reviews are warranted. We have reported that major change initiatives and cultural changes take time to fully implement and take effect. In our 2003 report on the Architect of the Capitol, for example, we reported that the experiences of successful major change management initiatives in large private and public sector organizations suggest that they can often take at least 5 to 7 years until they are fully implemented and the related cultures are transformed in a sustainable manner. Additionally, we reported that fundamental changes in the Architect of the Capitol’s culture will require a long-term, concerted effort. The same may be true for CSA; much about CSA is new and, given the nature of this type of cultural transformation, it may simply take time for staff to adjust to the new paradigm. To address this issue, FMCSA, among other things, is using the pilot states as mentors for the states that did not participate in the pilot test, invited participants from pilot-test states to describe the new process to their peers in non-pilot states, and has put CSA on the agenda of annual in- service training sessions. Additionally, FMCSA plans to develop a systematic change management plan.

As we have previously discussed, several steps and issues remain before FMCSA can fully implement CSA carrier oversight activities. Specifically, FMCSA has not completed a key technology to fully implement the interventions and provided training on interventions yet to be implemented,  developed and issued the NPRM to take action against unfit carriers based on CSA data,  addressed staffing issues and completed efforts to help staff shift to a new safety enforcement paradigm.

FMCSA officials acknowledged delays in implementing CSA’s carrier oversight activities and the need to complete key tasks and address certain issues. However, they maintain that delays are to be expected when implementing a major program such as CSA and that, in their opinion, FMCSA has implemented the bulk of CSA’s oversight activities. They acknowledged that risks associated with FMCSA’s ability to complete these items and address budgetary issues could affect their ability to fully implement CSA, as well as CSA’s effectiveness, and noted that they track open issues and the associated risks and mitigation strategies. Although FMCSA officials indicated they have periodically briefed congressional staff of their progress in developing and implementing CSA, FMCSA has not developed any type of comprehensive document that specifically outlines its status, implementation delays, and other issues that need to be addressed, or identifies the risks associated with these problems and strategies to mitigate them.

Our past work has shown that the early identification of risks and strategies to mitigate them can help avoid negative outcomes when implementing large-scale projects. For example, in our 2010 report examining the Federal Railroad Administration’s (FRA) efforts to implement a Positive Train Control (PTC) system, we reported that uncertainties about tasks, such as potential delays in developing PTC components, software, and subsequent testing and implementation of PTC systems, raise certain risks to successfully completing PTC on time. Specifically, potential delays in developing PTC components, software, and subsequent testing and implementation of PTC systems, raise the risk that railroads will not meet the implementation deadline and that the safety benefits of PTC will be delayed. We noted that FRA officials were aware of some of these risks, but said it was too early to know whether they were significant enough to jeopardize successful implementation. However, we also noted that, as FRA moves forward with monitoring railroad’s implementation of PTC, the agency will have more information regarding the risks to completing PTC on time and would thus be in a better position to inform Congress and other stakeholders of the risks and mitigation strategies associated with implementing the system. Similarly, our 2004 report examining an Amtrak project to manage improvements to the Northeast Corridor noted that early identification and assessment of problems would allow for prompt intervention, increasing the likelihood that corrective action could be taken to get the project back on track. Risk identification and management are also essential in the case of CSA, which FMCSA developed with the goal of significantly improving motor carrier safety. Regularly reporting information on what steps FMCSA needs to complete in order to implement CSA—including a timetable—as well as the risks and mitigation strategies associated with not completing each step or addressing each issue, would put FMCSA in a better position to respond to problems when they occur and thus better ensure that FMCSA could complete CSA’s implementation as planned. This would also provide Congress and other stakeholders with important information as to FMCSA’s status in implementing CSA and the associated risks, which would help Congress make decisions about the program.

FMCSA Has Made Little Progress on Driver Fitness Ratings

Although the implemented CSA oversight activities have provided FMCSA additional tools to provide information on drivers and assess their safety performance, FMCSA has only recently begun steps to develop the process to separately rate the safety fitness of all drivers under CSA. Since CSA’s initiation, FMCSA has prioritized implementation of the carrier oversight activities. FMCSA is seeking to clarify its authority to prohibit individual drivers, if determined to be unfit based on ratings, from operating in interstate commerce. FMCSA officials believe that arguably the agency currently has this authority, but acknowledge that seeking clarification from Congress would be prudent. FMCSA is seeking this authority as part of the next surface transportation reauthorization and has provided committees of Congress technical legislative drafting assistance to this effect.

FMCSA officials also explained they now have access to more information on drivers than they previously had so that implementing the driver component is not as critical to CSA’s ability to improve safety as they believed when designing the program. For example, the Unsafe Driver BASIC provides additional oversight of drivers and allows FMCSA to address unsafe driver behaviors by intervening with carriers that employ unsafe drivers. Other systems also now allow FMCSA to evaluate drivers:  The Driver Safety Measurement System (DSMS) uses safety data from roadside inspections and crashes to measure drivers’ safety in a manner similar to that used under SMS and allows FMCSA and state partners to identify unsafe, or “red flag,” drivers. The red flag driver investigation process examines drivers receiving certain violations during the course of motor carrier investigations. However, since FMCSA has not implemented driver safety fitness determinations, the agency only uses DSMS internally and for law enforcement purposes.  The Pre-Employment Screening Program (PSP) allows carriers to view 5 years of individual drivers’ crash data from FMCSA’s MCMIS as well as 3 years of roadside violation data from MCMIS. Although PSP provides useful information, it was not intended to be a comparative tool and thus does not allow carriers to determine how safe or unsafe a driver is compared to other drivers. Also, participation in PSP is voluntary; motor carriers must pay a subscription fee for this service.

Nonetheless, including a fitness determination would expand FMCSA’s oversight by measuring individual driver performance and systematically identifying unsafe commercial drivers for safety enforcement. It would allow carriers to determine an individual driver’s safety relative to other drivers and increase the usage of driver safety data among the motor carrier industry. FMCSA’s 2005 study of large truck crashes found that driver behavior is the single largest cause of crashes. FMCSA officials indicated that they still plan to assess driver fitness as part of CSA but have not developed a plan or set any timetable for doing so. FMCSA has also not determined how driver safety determinations will be used or assessed the safety risk of delayed implementation of them.

CSA has the Potential to Improve Safety by Identifying More At- Risk Carriers, but the Program’s Full Impact Nationwide Is Unclear

CSA has the potential to identify higher-risk carriers under more precisely defined areas of safety performance, and FMCSA has an expanded range of interventions to follow up with them. Collectively, these changes offer the potential to improve safety. However, not all carriers are inspected, and larger sized motor carriers are likely to have more inspections and thus, more likely to be ranked under SMS than smaller sized motor carriers. Moreover, the technology FMCSA has developed to select carriers for inspection did not allow inspectors we observed to quickly determine if a carrier’s past history warranted an inspection. Instead, they used it to identify what needed to be inspected once a carrier was already selected for inspection. As a result, some states use other technologies that incorporate FMCSA’s system, or other methods to select carriers that may not be systematic. Furthermore, until FMCSA completes new performance metrics, gauging the extent to which CSA improves safety will be problematic.

CSA Has Potential to Improve Safety through Better Use of Data and More Contact with Carriers

To improve safety, CSA makes better use of roadside inspection data in the following ways:  SMS makes greater use of the data available from roadside inspections than SafeStat did. Under SafeStat, only out-of-service violations and selected moving violations were used for estimating carriers’ scores under the Driver and Vehicle safety evaluation areas. In SMS, any violation found is used in calculating a carrier BASIC score. This should help FMCSA to improve overall safety by allowing it to identify carriers with recurring types of safety violations that may have been missed under the prior SafeStat system.  SMS allows for more precision in the measurement of safety, since, as we discussed previously, the BASIC scores and Crash Indicator measure carrier performance in seven areas, rather than the four used under SafeStat. For example, CSA measures driver performance at the motor carrier company level in several categories, including unsafe driving, fatigued driving, driver fitness, and the use of controlled substances and alcohol, whereas SafeStat calculated an overall rating based on all these driver factors combined. This breakdown not only allows for a more precise determination of motor carrier safety performance overall but also allows FMCSA to better identify specific areas of safety shortcomings. For example, CSA can indicate if a carrier is having a problem with driver fatigue, whereas SafeStat could not provide this level of detail. Thus, interventions can be targeted to the specific area of safety concern.  SMS creates percentile ranks for carriers within each BASIC and in the Crash Indicator, rather than producing just one total summed score, as SafeStat did. Thus, SMS has the potential to improve safety by reporting scores on the separate areas of safety problems and making carriers’ performance in this area explicit. For example, CSA can indicate that, although a carrier has a relatively poor ranking in the Cargo-Related BASIC, the carrier has a good ranking in the Unsafe Driving indicator, thereby enabling FMCSA to focus its interventions on carrier practices that have the greatest impact on safety.

SMS also allows FMCSA to conduct interventions with a greater number of motor carriers. SMS identifies about 45,000 motor carriers each month that exceeded the thresholds in one or more BASICs or the Crash Indicator. By comparison, under SafeStat, a similar number of about 45,000 carriers per month were identified as exceeding the threshold on one or more safety evaluation areas to varying degrees, on a scale of A to G. However, under SafeStat, only those carriers with a SafeStat rating of A, B, or C were prioritized for SafeStat’s intervention—a full compliance review—resulting in a smaller percentage of motor carriers with an identified safety problem receiving the intervention. For example, during all of fiscal year 2009, 16,512 compliance reviews were carried out by FMCSA and state partners on motor carriers rated under SafeStat. Under CSA, any carrier exceeding a threshold in even one BASIC or in the Crash Indicator will receive an intervention of some type.

The reason FMCSA can contact carriers with a wider range of violations—including less severe violations—than it did under SafeStat is that CSA provides a wider range of intervention tools, some of them requiring few resources to implement. CSA’s range of interventions—from the resource-intensive On-site Comprehensive Investigation to the relatively low-resource Warning Letter—provide FMCSA with more tools for contacting carriers, calibrating the intervention to the severity of the violation. Under CSA, all carriers newly identified as exceeding the threshold in one or more safety areas in a given month are subject to some type of safety intervention by FMCSA, most commonly a Warning Letter. During the first 6 months of fiscal year 2011, FMCSA sent 19,470 Warning Letters and, along with state partners, conducted 3,190 CSA On- site Focused Investigations in addition to completing 5,684 compliance reviews through May of 2011, for 28,344 total safety interventions. Preliminary evidence from the pilot test suggests that even the warning letters have an effect on safety. Twelve months after receiving only a Warning Letter, 17 percent of test carriers exceeded at least one SMS threshold as opposed to 45 percent of the control carriers who did not receive Warning Letters. Reaching more carriers with enforcement actions should enable FMCSA to improve safety.

Not All Carriers Have Enough Inspections to Receive Safety Rankings

While the pilot test suggests that SMS has the potential to improve safety over the prior SafeStat system, SMS’s ability to calculate BASIC scores for carriers is dependent upon sufficient roadside inspection data for that carrier, which are not always available for a significant segment of carriers. Analysis of the data from the pilot test states found that a substantial proportion of motor carriers lack sufficient data for ranking in the six BASICs and Crash Indicator: Specifically, the Fatigued Driving and Unsafe Driving BASIC both require a minimum of three relevant inspections and at least one relevant violation for a motor carrier over the past 24 months; the Vehicle Maintenance, Driver Fitness, and Cargo- Related BASICs each require a minimum of five relevant inspections and at least one relevant violation over the preceding 24 months. Table 5 shows the percentage of carriers in the pilot test states that have sufficient data for ranking in a BASIC or the Crash Indicator.

While most large motor carriers have enough data to be considered and rated under SMS, the majority of smaller carriers do not. For example, about 48 percent of carriers with 51 to 500 vehicles and about 71 percent of carriers with 501 or more vehicles have sufficient ranking in the Unsafe Driving BASIC but only about 1 percent of carriers with 5 or fewer vehicles do. The majority of companies in operation are small motor carriers with 5 or fewer vehicles; the lack of sufficient data for ranking on a BASIC is greatest in this segment of the carrier fleet. Those carriers with 2 or fewer roadside inspections are only potentially ranked by SMS through the Controlled Substances/Alcohol BASIC or the Crash Indicator. Those with 3 to 4 inspections are below the minimum data sufficiency requirements for the Vehicle Maintenance and Driver Fitness BASICs. This data limitation will continue to prevent the SMS from functioning at full capability until efforts to expand roadside inspection measurement coverage across the motor carrier fleet succeed. In the meantime, the effect of this data sufficiency limitation is that safety ranking by SMS is more concentrated among the large sized motor carriers than it is among the more numerous smaller sized motor carriers.

Based on visits to inspection stations and interviews with inspection officials in eight states, we found that not all states use methods that systematically select trucks for roadside inspections, which can limit CSA’s ability to improve motor carrier safety. FMCSA provides all states with its Inspection Selection System (ISS-2010) software, designed to systematically identify carriers with known poor safety performance. Vehicle selection methods that factor in safety performance offer more assurance that roadside inspections will ultimately prevent crashes by focusing resources on higher-risk carriers. The ISS-2010 software also systematically identifies carriers that have not been ranked in any of the BASICs by the SMS, so that inspectors can inspect those carriers’ trucks to determine their compliance. Because of the pace at which trucks move through the scales at inspections stations, however, inspectors we observed rarely had time to access the ISS-2010 on FMCSA’s website before deciding which trucks to inspect. Thus, inspectors mainly used ISS-2010 to obtain information about trucks that have already been selected for inspection by other means (see below).

Many states use software that allows inspectors to bypass some low-risk trucks from the inspection station, thus allowing them to select carriers for inspection from a group with a history of safety problems or unknown safety performance. These third-party software products incorporate the ISS-2010 algorithms to allow trucks belonging to carriers with good safety performance to bypass inspection stations. When these trucks are allowed to bypass the weigh station, inspection resources can be expended on carriers with riskier or unknown safety performance, according to the software. For example, in 30 states, inspectors rely on a product called PrePass. PrePass incorporates the ISS-2010 selection algorithms with other proprietary criteria to gauge a carrier’s safety performance, including crash risk, before its truck enters an inspection station. Carriers that participate in PrePass receive transponders for their trucks; weigh stations are fitted with equipment that receives signals from the transponders. The transponder sends a signal to the inspection station that alerts inspectors as to whether the participating truck can bypass the inspection station or if it must come in for an inspection. All nonparticipating trucks must enter the station when it is open. Inspectors in states that use such software products then employ a combination of other methods, some noted below, to select trucks for inspection from among those that enter the weigh station. FMCSA officials stated that states are encouraged to use federal roadside inspection grant funds to purchase technology to assist their inspectors in systematically selecting trucks for inspection. FMCSA currently does not require states to use ISS-2010 software or products like PrePass, although it encourages them to do so.

While some of the selection methods we observed being used at inspection stations take some aspects of crash risk into account, none are as systematic as would be the case if inspectors were able to use the ISS-2010 algorithms for truck selection. Some also may, by chance, select for inspection the trucks of carriers previously unranked by CSA, thereby broadening the base of carriers the SMS can potentially rank. We observed the following selection methods:  Weight as an initial selection factor. All trucks entering a weigh station will be weighed. If the overall weight or individual weight on a particular axle exceeds the allowed weight, inspectors put the truck and driver out of service. This method addresses safety performance and may by chance select trucks of carriers previously unranked by the SMS because all trucks must cross the scales. The truck may not resume its journey until its weight issues are resolved, and the inspector has discretion to conduct further inspection.  Obvious problems. When inspectors notice obvious problems related to safety performance on a truck as it moves across the scale—such as a flat tire, unattached hoses, incorrect or damaged placards, etc.— they may pull the vehicle over for inspection. Many of these problems could involve safety performance issues and may result in selecting trucks of carriers previously unranked by the SMS.  Random selection. Inspectors choose trucks for inspection randomly from among those not put out of service for weight issues. This method does not gauge crash risk or other aspects of safety performance but could select trucks of carriers previously unranked by the SMS.  Local discretion. These methods may focus inspectors’ efforts on particular types of inspections, carriers, trucks, or loads for a period of time. Local discretion selection methods can be guided by the certification level of inspectors available at the station, the training needs of those inspectors, or news stories about crashes of particular types of vehicles or loads, among other things. In some cases, inspectors may focus their efforts on factors that influence safety, perhaps in response to public opinion about the safety performance of particular types of vehicles or loads. These methods could also result in selecting trucks of carriers previously unranked by the SMS.

All of these methods are limited in identifying higher-risk trucks and carriers. For example, a truck belonging to a carrier with a history of driver fatigue issues would not be readily identifiable to an inspector unless a software product employing the ISS-2010 BASIC-supported algorithm flagged it.

No inspection selection method can assist weigh station inspectors in selecting trucks if drivers avoid the weigh station entirely. Our observations at state inspection stations and discussions with inspection officials revealed that some drivers attempt to evade roadside inspection in different ways, allowing some carriers to potentially operate entirely beyond the scope of CSA. For example, drivers may avoid driving past a weigh station during its regular hours of operation. Inspection facilities in many states are open limited hours, and state officials told us there is a significant level of truck traffic when stations are closed. Because of physical or staffing constraints at some weigh stations, we observed that staff may close a station periodically during its standard hours of operation to relieve crowding or avoid back ups of trucks that could present a safety hazard on the freeway. State police and other officials in a number of states also indicated that budgetary constraints may force them to reduce weigh stations’ hours of operation, decreasing the number of trucks they can inspect and increasing the travel-time flexibility of drivers seeking to avoid inspection. State police officials also told us that some drivers seek to evade inspection by pulling over to the side of the road until a station closes or by altering their routes to drive around weigh stations, either on other highways or on smaller roads, sometimes within sight of staff at the weigh station. Depending on resources available at the station, troopers may or may not be able to leave the station to stop drivers whose trucks should be inspected. According to a number of state inspection officials, when inspectors do inspect trucks of drivers seeking to avoid inspection, they often find serious safety violations, reiterating the potential importance of appropriately targeting inspection resources to road safety.

Until FMCSA Implements New Performance Measures, Gauging Extent of CSA-related Safety Improvement Will Be Problematic

FMCSA has begun to develop performance measures to assess CSA’s nationwide performance in improving safety, but has not yet set a timetable for their completion. Indications that CSA may improve safety exist. Specifically, the UMTRI evaluation of the pilot test indicates that CSA’s SMS and new, expanded set of interventions increased FMCSA’s ability to improve safety in the four pilot states. However, performance measures are needed to gauge the effectiveness of CSA in improving safety as it is implemented nationwide.

We have previously reported that agencies need to set quantifiable outcome-based performance measures for significant agency activities, such as CSA, to demonstrate how they intend to achieve their program goals and measure the extent to which they have done so. Performance measures allow an agency to track its progress in achieving intended results, which can be particularly important in the implementation stage of a new program. Performance measures can also help inform management decision making, such as the need to redirect resources or shift priorities. In some of our prior work we have recommended that agencies develop methods to accurately evaluate and measure the progress of implementation, and develop contingency plans if the agency does not meet its milestones to complete tasks. In addition, performance measures can be used by stakeholders, such as state law enforcement partners, carrier associations, and the public who use the nation’s highways, to hold FMCSA accountable for results. With performance measures, FMCSA Divisions and state partners will be able to set priorities and measure results by state or overall.

FMCSA has been working on developing performance measures for CSA results and program implementation progress. FMCSA has proposed several performance measures for CSA, but they have not yet been approved within the agency. Two of the proposed measures would assess outcomes of CSA. The first would determine the number of carriers that received a specific CSA intervention in 1 year and then showed improvement in the next year. The second would measure the level of compliance from all inspections in a baseline year before CSA was implemented (e.g., 2007) and compare that level against compliance in subsequent years to quantify improvements in compliance across the entire industry. FMCSA is also considering output measures, such as the increase in the number of carriers reviewed once off-site and focused investigations are fully implemented. According to FMCSA officials, these proposed measures have not yet been approved by the Administrator, and implementation will depend on accumulating relevant CSA intervention data once the carrier oversight activities are fully deployed in 2012, as expected. Under this timeline, 2012 would become the baseline year, which means 2013 would become the first year in which FMCSA could begin to develop CSA performance targets such as the percentage of carriers that showed safety improvements after being subject to CSA interventions.

FMCSA has also begun efforts to track its progress in implementing CSA. FMCSA has identified the specific steps it has taken to implement CSA, as well as the states in which the various CSA oversight activities have been implemented (i.e., pilot states vs. nonpilot states, and, for those oversight activities that have not been implemented, when FMCSA plans to implement them). When ultimately developed and implemented, such measures will help provide CSA managers with information on the status of CSA implementation and allow them to make adjustments, if necessary, to meet established timeframes.

Conclusions

FMCSA’s CSA program has been partly implemented and shows the potential for improving motor carrier safety. However, key aspects of the initiative, including using safety data from the new SMS system to take unsafe carriers and drivers off the road and enforcing other safety regulations, are indefinitely delayed. In the case of drivers, the plan for when and how to determine a driver’s fitness to operate vehicles based on the new measurement system has yet to be developed, and the safety implications of delayed implementation of drivers’ fitness ratings for FMCSA’s current goals to improve safety are unclear. FMCSA has also encountered several problems during implementation, including delays in developing technology needed for new interventions, and resistance from staff to shift to a new paradigm of more focused and less time-consuming reviews of carrier operations. Further, FMCSA has not established a process for regularly reporting to Congress and the public on CSA’s status, problems it has encountered in implementing CSA, the risks they pose to full implementation and its strategy for mitigating these risks. This type of information is essential to assist Congress in making decisions about funding or authorizations for the program and assure Congress and stakeholders that CSA is being successfully implemented. To this end, FMCSA has made progress in developing performance measures for determining the extent to which investigative staff are using new CSA interventions and the safety outcomes of these interventions. However, until these measures are completed and are being implemented, the extent of CSA’s effectiveness in improving safety will remain unclear to FMCSA management, Congress, and the public.

Recommendations for Executive Action

We recommend that the Secretary of Transportation direct the FMCSA Administrator to take the following two actions  develop a plan for implementing driver fitness ratings that prioritizes steps that need to be completed and includes a reasonable timeframe for completing them. The plan should also address the safety implications of delayed implementation of driver fitness ratings. regularly report to Congress on CSA’s status; the problems that FMCSA has encountered during the implementation of CSA and the risks they pose to full implementation of CSA; its strategy for mitigating these risks; and a timetable for fully implementing CSA and reporting the progress made in developing and implementing CSA performance measures.

Agency Comments

We provided a draft of this report to the Department of Transportation for review and comment. The Department did not agree or disagree with our recommendations but said it would consider them. The Department provided technical comments and clarifications, which we incorporated as appropriate. At a meeting on September 23, 2011, to discuss the Department’s comments, FMCSA officials confirmed that they intend to continue implementing the driver fitness ratings. Previously, FMCSA officials indicated that they were considering implementing these ratings but had made no final decision. In response to this new information, we modified the language of our recommendation regarding driver fitness ratings. Our recommendation originally focused on having FMCSA determine the safety implications of not fully and expeditiously implementing the driver fitness ratings and, if it determined that full implementation was necessary, to then develop an implementation plan. To reflect that FMCSA has decided to proceed with implementing the driver fitness ratings, we modified our recommendation to focus instead on an implementation plan.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Transportation, the Administrator, Federal Motor Carrier Safety Administration; and the Director, Office of Management and Budget. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions regarding this report, please contact me at (202) 512-2834 or at flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

GAO was directed by a 2009 Senate Committee Report, adopted by the conference committee, to conduct a study as part of the continued monitoring of the Compliance, Safety, and Accountability program (CSA) implementation. Specifically, this report addresses (1) the status of the CSA rollout and what issues, if any, could affect the full and effective implementation of the program and (2) CSA’s potential to improve safety.

To determine the status of the CSA rollout and challenges that could affect the full implementation of the program, we analyzed Federal Motor Carrier Safety Administration (FMCSA) documentation, including information on FMCSA’s website (www.fmcsa.dot.gov), periodic outreach e-mails from CSA program officials, and CSA training materials. Additionally, we reviewed congressional testimony provided by FMCSA’s Administrator. We also reviewed an evaluation of the pilot test conducted by the University of Michigan’s Transportation Research Institute (UMTRI). We interviewed FMCSA and National Transportation Safety Board headquarters’ officials as well as national representatives of carrier industry associations (see table 6)

We also attended the Commercial Vehicle Safety Alliance (CVSA) annual conference in September 2010 and interviewed representatives of several State Partners to discuss CSA implementation, as indicated in table 7. We also attended two FMCSA-sponsored outreach sessions discussing different aspects of CSA and carrier motor vehicle safety, one from the carrier’s perspective.

Additionally, we visited eight states (four that participated in the pilot program: Georgia, Maryland, Minnesota, and Missouri, and four that did not: California, Mississippi, Texas, and Utah) to interview FMCSA Division and State Partner officials as well as industry groups and some carriers. (See table 8 for criteria we used to select these states.) We collected and reviewed other CSA implementation and background documentation during these visits. (See table 9 for agency and industry organizations we interviewed during state visits.)

To obtain information on motor carriers’ knowledge of and experiences with CSA, we selected a nongeneralizable random sample of motor carriers from the Motor Carrier Management Information System (MCMIS) carrier census file and conducted brief, structured telephone interviews. We screened the population from which we selected the sample to remove foreign carriers and those carriers that had not updated their census (MCS-150) forms with FMCSA in the prior 2 years. We divided the carriers into three size categories—small, medium, and large—based on the number of vehicles associated with the company and randomly selected a group of carriers within each size category to participate in the structured interviews. During the interviews, we asked about the interviewees’ knowledge and understanding of CSA; interviewees were owners, safety managers, or others who would have knowledge of a carrier’s safety practices and performance. We obtained responses from 55 motor carriers out of the 270 we attempted to contact.

To determine CSA’s potential to improve safety, we analyzed FMCSA documents describing the design and function of the Safety Measurement System (SMS), how the severity of violations were weighted, and other design documentation as it was released, particularly comparing the SMS with SafeStat. We also analyzed UMTRI’s pilot test study findings. We reviewed UMTRI’s statistical methodology and its reliability assessment of the FMCSA data used for the study and determined that the results of UMTRI’s pilot evaluation study were sufficiently reliable for our purposes. We obtained a copy of the May 2011 MCMIS inspection data, upon which five publicly available Behavior Analysis and Safety Improvement Categories (BASIC) scores for carriers were based, and analyzed it to determine the extent to which motor carriers lacked a sufficient number of roadside inspections for measurement under the BASICs in SMS. We electronically tested the data for completeness and coding accuracy, and found it sufficiently reliable for the purposes of our engagement.

We also analyzed the function of FMCSA’s Inspection Selection System software, which is designed to select trucks for inspection and thereby guides data collection for the SMS. We did not model the SMS in order to test its function ourselves, as it was modified several times during the course of our review. During our state visits, we also visited weigh stations or other truck inspection sites to interview inspectors about how they select trucks for inspection and how CSA has affected their work, and the data they obtain during inspections. We observed truck inspections during these visits. We also obtained information on crash data quality by analyzing studies UMTRI conducted on states’ MCMIS crash data reliability as well as FMCSA’s publicly available crash data evaluation tools.

We conducted this performance audit from June 2010 to September 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Susan Fleming (202 512-2834)

Staff Acknowledgments

In addition to the individual named above, Ed Laughlin, Assistant Director; Lynn Filla-Clark, Analyst-in-Charge; Carl Barden; Lauren Calhoun; Alison Hoenk; Delwen Jones; Elke Kolodinski; Kirsten Lauber, Sara Ann Moessbauer; Rebecca Rygg, Amy Rosewarne; and Larry Thomas made key contributions to this report.